about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-11 15:22:07 +0000
committerPjotr <wrk@bosa.lan>2026-11-11 15:22:07 +0000
commit8a0b4059114d2628fe3362c816efcfc4262fc897 (patch)
treee095c0e348ad124efa0c05461f2c71be0932db0f /docs
parentee007a3731d186dc12c1b8b81589a03e8eb95acb (diff)
downloadgenenetwork4-8a0b4059114d2628fe3362c816efcfc4262fc897.tar.gz
docs: why the two engines' h2 disagree, and the three measurements that settle it
Written down from a real analysis (BXD.8, 35 phenotyped animals): trait
16769 has both engines saying kinship dominates (BulkLMM h2 = 1.000,
sigma2_e = 0; GEMMA clamps l_mle on every marker but the chr2 peak), while
16627 is a genuine disagreement (h2 = 0.000 against a clamped l_mle
everywhere). Both sit on a boundary, which is expected at n = 35 -- but
opposite boundaries mean the engines are estimating different things:

  * different kinship matrices (GEMMA's own relatedness from the BIMBAM
    dosages, one per chromosome under LOCO, against calcKinship's
    2*(X-0.5)(X-0.5)'/p + 0.5 with a unit diagonal);
  * different encodings (0/0.5/1 with NA, against the same dosages with
    missing values mean-imputed, against R/qtl's two-of-three codes) --
    and 16627 is nearly determined by the chr2 genotype;
  * different variance-component step (once under the null against
    per-chromosome LOCO, clamped differently at a boundary).

The three measurements that would settle it, with what is known about each:

  1. GEMMA's pve: it prints "pve estimate" when run with -pve, and
     gemma-wrapper keeps 223 per-chromosome .log.txt files under its cache
     dir -- none contain pve, because the flag is not passed. One argument
     on the GWA step plus a parser.
  2. BulkLMM's profile over an h2 grid: BulkLMM.profile_LL(y::Matrix, G,
     covar, K, h2_grid, markerID; reml) is internal (not exported) and
     returns (ll_list_null, ll_list_alt); it wants a one-column phenotype
     matrix and a covariate matrix. An attempt to call it from the driver
     is not landed -- the signature mismatch is the whole of the remaining
     work.
  3. The kinship matrices, or a summary: calcKinship is one line and can be
     written per panel, GEMMA's is already cached as *.k.json; correlating
     off-diagonals says whether the engines disagree about relatedness
     before arguing about variance components.

Until those exist, the reading in the analysis holds: BulkLMM's h2 and the
thresholds that rest on it are approximate on a boundary, and GEMMA's
16627 p-values come from a kinship-dominated model.
Diffstat (limited to 'docs')
-rw-r--r--docs/bulklmm.md55
1 files changed, 55 insertions, 0 deletions
diff --git a/docs/bulklmm.md b/docs/bulklmm.md
index fbb3b37..21e7f48 100644
--- a/docs/bulklmm.md
+++ b/docs/bulklmm.md
@@ -58,6 +58,61 @@ the engine agrees with the record — and the peak clears the α = 0.01 threshol
   lod, L_perms)`; `SE` is not among them, so the reply's `se` column does not
   exist and the rows are `marker, chr, pos, lod, beta, h2`.
 
+## When the two engines' h2 disagree, and what settles it
+
+Reported from a real analysis (BXD.8, 35 phenotyped animals):
+
+| trait | BulkLMM h2 | GEMMA `l_mle` | reading |
+|---|---|---|---|
+| 16769 | 1.000, σ²_e = 0 | at the upper clamp on every marker except the chr2 peak | both say kinship dominates; the peak's drop is the marker absorbing the effect |
+| 16627 | 0.000, σ²_e = 0.402 | at the upper clamp everywhere, peak included | a genuine disagreement |
+
+Both estimates sit on a boundary, which is expected at n = 35 — the likelihood
+is flat there — so neither number is a precise proportion. But *opposite*
+boundaries on the same phenotype and the same genotypes means the two engines
+are estimating different things, and it is worth knowing which:
+
+1. **Different kinship matrices.** GEMMA computes its own relatedness from the
+   BIMBAM dosages (and, in LOCO mode, one per chromosome); BulkLMM's
+   `calcKinship` is `2·(X−0.5)(X−0.5)ᵀ/p + 0.5` with a unit diagonal. Two
+   matrices, two variance components — not comparable until their correlation is
+   known.
+2. **Different encodings.** GEMMA gets `0 / 0.5 / 1` dosages with `NA` for
+   missing; BulkLMM gets the same dosages with missing values mean-imputed per
+   marker, and R/qtl's `convert2riself` (two of three codes) is a third.
+   Trait 16627 is nearly determined by the chr2 genotype (B above 1.0, D below
+   0.6), so the residual variance differs a lot between those encodings.
+3. **Different variance-component step.** BulkLMM estimates once under the null
+   with `assumption = "null"` (and REML or ML); GEMMA optimises per chromosome
+   under LOCO. On a boundary both optimisers clamp, and they clamp to different
+   conventions.
+
+**What would settle it, in order of cost:**
+
+* **GEMMA's own variance components.** GEMMA prints `pve estimate` and
+  `se(pve)` in its `.log.txt` when it is run with `-pve`, and gemma-wrapper keeps
+  the per-chromosome logs under its cache directory (223 `.log.txt` files here,
+  **none** containing `pve` — the flag is not passed). The change is one
+  argument on the GWA step and a parser over those logs; then `/compute/gemma`
+  can report pve per chromosome, as `/compute/bulklmm` reports h2.
+* **BulkLMM's likelihood over an h2 grid.** The package has
+  `BulkLMM.profile_LL(y::Matrix, G, covar, K, h2_grid, markerID; reml)` (internal,
+  not exported), returning `(ll_list_null, ll_list_alt)` over the grid. It wants
+  the phenotype as a one-column matrix and a covariate matrix (an all-zero column
+  is what `scan` uses with no covariates). Printing that profile at the peak
+  marker says whether an h2 of 0.000 is the maximum or the flat edge of one — the
+  difference between "no heritability" and "the estimator cannot tell".
+* **The kinship matrices, or a summary.** BulkLMM's `calcKinship(g)` is one line
+  and can be written out per panel; GEMMA's is already cached as a `-gk`
+  `*.k.json`. Correlating their off-diagonals (or reporting min/median/max) is
+  enough to say whether the two engines disagree about *relatedness* before
+  arguing about what the variance components mean.
+
+Until then, the honest reading is the one the analysis reached: BulkLMM's `h2`
+and the thresholds that depend on it are approximate where `h2` is on a
+boundary, and GEMMA's p-values for 16627 come from a kinship-dominated model —
+its peak should not be compared with a fixed-effects LOD.
+
 ## Cores: `scan()` is serial by design
 
 This was measured, not assumed. `Threads.@threads` appears in `bulkscan.jl` and