about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-11 15:42:40 +0000
committerPjotr <wrk@bosa.lan>2026-11-11 15:42:40 +0000
commit54d704ab7a6ec6e39ccb66e08c2334652a437cfa (patch)
tree12fdfd2b3e56d5a6cdfdf9d65739b55c26b97cb9 /docs
parent48e33540c4a00f0077963c3410628a6258448558 (diff)
downloadgenenetwork4-54d704ab7a6ec6e39ccb66e08c2334652a437cfa.tar.gz
docs: GEMMA's pve is one flag away, not a dead end
Tested -lmm 9 with -pve on a synthetic BIMBAM trio (40 individuals, 60
markers, GEMMA 0.98.6): the flag is accepted and the log carries

  ## pve estimate in the null model = ...
  ## se(pve) in the null model = -nan
  ## vg estimate in the null model = 0
  ## ve estimate in the null model = 0

So what was missing is plumbing, not support. Two gemma-wrapper changes are
in its working tree (5eb53fa): the per-run logs are copied beside the
archive -- they were only inside the .tar.xz -- and a failing child's own
ERROR lines are printed before the Ruby backtrace. Once pushed and pinned,
/compute/gemma can pass -pve and report pve/vg/ve per chromosome beside
l_mle, which is what the 16627 and 16769 comparison needs.

Also recorded: on random data GEMMA returns se(pve) = -nan with vg = ve = 0,
so a boundary pve is a direction rather than a proportion -- the same
caveat as BulkLMM's h2, from the other engine.
Diffstat (limited to 'docs')
-rw-r--r--docs/bulklmm.md41
1 files changed, 23 insertions, 18 deletions
diff --git a/docs/bulklmm.md b/docs/bulklmm.md
index 83d4686..84ebe4d 100644
--- a/docs/bulklmm.md
+++ b/docs/bulklmm.md
@@ -89,24 +89,29 @@ are estimating different things, and it is worth knowing which:
 
 **What would settle it, in order of cost:**
 
-* **GEMMA's own variance components.** GEMMA prints `pve estimate` and
-  `se(pve)` in its `.log.txt` only when run with `-pve`, which our GWA step does
-  not pass. Whether `-pve` is *compatible* with `-lmm 9` under `--loco` is
-  **untested**, not refuted: a hand-made invocation with and without the flag
-  fails identically inside gemma-wrapper's `parallel` batch of per-chromosome
-  jobs, so the flag was never isolated. What the wrapper does not surface is the
-  failing job's own error — the batch's stderr goes to a temporary directory
-  that is removed — and that is the defect worth fixing first, because it hides
-  every GEMMA failure, not just this one. The logs the wrapper keeps are the
-  *kinship* step's — 223 `.log.txt` files under `/tmp/gn4-gemma`, none with a
-  `pve` block, because the per-chromosome LMM runs write theirs inside
-  gemma-wrapper's temporary directories and they are not retained. Two routes
-  remain, neither landed: (a) run GEMMA ourselves for the null fit — `-gk` to
-  write a text K matrix, then `-lmm 1 -pve` on a single marker, both seconds —
-  and read `pve estimate in the null model` from its log; (b) ask gemma-wrapper
-  to forward `-pve` (it is not in its allowlist). (a) needs the kinship as plain
-  text: the wrapper caches it as JSON under a per-instance cache directory, and
-  the two attempts at writing it out by hand were not completed.
+* **GEMMA's own variance components — available, one flag away.** Tested on a
+  synthetic BIMBAM trio (40 individuals, 60 markers) with GEMMA 0.98.6: `-lmm 9
+  -pve` is accepted and writes the block
+
+      ## pve estimate in the null model = 1.62846e-06
+      ## se(pve) in the null model = -nan
+      ## vg estimate in the null model = 0
+      ## ve estimate in the null model = 0
+
+  So the flag works with `-lmm 9`; what was missing is the plumbing. Two
+  wrapper changes are in `gemma-wrapper` (5eb53fa, awaiting push and a pin
+  bump): the per-run `.log.txt` files are copied **beside** the archive (they
+  were only inside the `.tar.xz`, so reading `pve` meant unpacking a
+  multi-megabyte archive), and a failing child's own `ERROR`/`****` lines are
+  printed before the Ruby backtrace. With those, `/compute/gemma` can pass
+  `-pve` on the GWA step and read `pve`/`vg`/`ve` per chromosome out of
+  `<hash>.<chr>.cXX.txt.log.txt` — and report them beside `l_mle`, which is what
+  the 16627/16769 comparison needs.
+
+  Note what GEMMA itself returned on random data: `se(pve) = -nan` with
+  `vg = ve = 0`. On a boundary GEMMA's own standard error is not a number, so a
+  boundary pve is a direction, not a proportion — the same caveat as BulkLMM's
+  `h2`, from the other engine.
 * **BulkLMM's likelihood over an h2 grid.** The package has
   `BulkLMM.profile_LL(y::Matrix, G, covar, K, h2_grid, markerID; reml)` (internal,
   not exported), returning `(ll_list_null, ll_list_alt)` over the grid. It wants