about summary refs log tree commit diff
path: root/docs/bulklmm.md
AgeCommit message (Collapse)Author
7 daysgemma: the pve note carries every chromosome, and the log carries the numbersPjotr
The note reported the range and named the two chromosomes that bounded it, which is not enough to answer the question the numbers exist for. Under --loco each pve belongs to a *different* kinship matrix, and GEMMA's per-chromosome .log.txt files live on the server -- so a caller told "0.31 to 0.95" could not ask which chromosome was which, or compare a single chromosome against BulkLMM. The note now lists all twenty, compactly, with the range and the two extremes keeping their vg/ve: GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451, chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to 0.9549 over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg 23.3659, ve 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933, ve 0.8661). and streaming::append_lines puts the unrounded values in the run log as variance: chr7 pve=0.954938 se_pve=0.357032 vg=76.0933 ve=0.86612 so a ?id= run's /compute/gemma/status stream carries them while the run is still in flight, rather than only in a reply someone has to keep. Both measured on BXD::10001 through the dev server before committing. The values are printed with four significant figures (scientific when the magnitude needs it), unchanged: rounding variance components to four decimals was the earlier bug, and these are the same numbers a reader compares with BulkLMM's h2. Docs: README's /compute/gemma section, SKILL's gemma bullet ("quote per-chromosome values, not an average: they are twenty different kinship matrices") and docs/bulklmm.md's example.
7 daysdocs: the two engines agree on heritability -- measured, not assertedPjotr
Once both engines' variance components were readable, BXD::10001 (34 animals) gives: BulkLMM, global null h2 = 0.819 (sigma2_e 5.381) GEMMA, LOCO per chr pve 0.3125-0.9558, median 0.8015, mean 0.7769 0.80 against 0.82. The engines never disagreed about the variance split; they disagreed about which chromosome's kinship to hold the rest of the genome out from. That closes the question this comparison started with. Also says how to quote them: GEMMA's pve with its range because it is per chromosome, BulkLMM's h2 as one number because it is global, and neither set of p-values against the other.
7 daysdocs: how pve gets deployed, and what the deployed box runsPjotr
GEMMA_WRAPPER_CMD is the lever: the profile's gemma-wrapper may be older than the pin, and pointing the variable at the store path is enough -- -pve reaches GEMMA either way, but the per-run .log.txt files where pve lives are only kept beside the archive from 5eb53fa on. Recorded the verified production state too: both processes restarted with gemma-wrapper-1.00-pre1-4.5eb53fa, and POST /compute/gemma for BXD::10001 on the running instance returns pve 0.3125 to 0.9558 over 20 chromosomes -- measured there, not on a fixture.
7 daysgemma: report GEMMA's own pve for the null modelPjotr
The GWA step now passes -pve, and the reply gains a note with the variance components GEMMA computed for the null model. Under --loco there is one per chromosome, so the note gives their range and names the two that bound it: GEMMA -pve, null model, per chromosome (LOCO): pve 0.3118 to 0.9549 over 20 chromosomes -- lowest 8 pve 0.3118 +/- 0.5009 (vg 23.3659, ve 12.3389), highest 7 pve 0.9549 +/- 0.3570 (vg 76.0933, ve 0.8661). That is BXD::10001 measured, not assumed. pve is the proportion of variance the kinship explains -- the same quantity as /compute/bulklmm's h2, per chromosome rather than genome-wide -- which is what the 16627/16769 comparison against BulkLMM needed. Reading the numbers took three decisions worth stating: - Four decimals is wrong for variance components. The first live run reported 'vg 0.0000, ve 0.0000' for a BXD::16769 null fit whose real components are 3.411e-5 and 3.411e-10 -- both rounded to zero, which reads as 'no variance' when the split is 1.0. Components are printed with four significant figures, scientific when the magnitude needs it, the way GEMMA's own logs do with %g. - se(pve) is -nan on a boundary, and Rust parses -nan as a NaN f64, so the numeric parse alone let it through as NaN. Non-finite values are now 'not available': a boundary estimate has no standard error. - Twenty chromosomes of values bury the other notes, and every value is in the .log.txt beside the archive anyway, so the note names the ends. Requires gemma-wrapper 5eb53fa, which keeps each run's .log.txt beside its archive instead of only inside the .tar.xz. Docs: README's /compute/gemma section, SKILL's gemma bullet (compare pve with h2, never vg with sigma2_e -- pve is scale-free), and docs/bulklmm.md's 'what would settle it', which now says the plumbing is in rather than pending.
7 daysdocs: GEMMA's pve is one flag away, not a dead endPjotr
Tested -lmm 9 with -pve on a synthetic BIMBAM trio (40 individuals, 60 markers, GEMMA 0.98.6): the flag is accepted and the log carries ## pve estimate in the null model = ... ## se(pve) in the null model = -nan ## vg estimate in the null model = 0 ## ve estimate in the null model = 0 So what was missing is plumbing, not support. Two gemma-wrapper changes are in its working tree (5eb53fa): the per-run logs are copied beside the archive -- they were only inside the .tar.xz -- and a failing child's own ERROR lines are printed before the Ruby backtrace. Once pushed and pinned, /compute/gemma can pass -pve and report pve/vg/ve per chromosome beside l_mle, which is what the 16627 and 16769 comparison needs. Also recorded: on random data GEMMA returns se(pve) = -nan with vg = ve = 0, so a boundary pve is a direction rather than a proportion -- the same caveat as BulkLMM's h2, from the other engine.
7 daysdocs: -pve is untested, not refutedPjotr
The earlier note said passing -pve crashes gemma-wrapper. It does not -- my hand-made invocation fails identically with and without the flag, inside the parallel batch of per-chromosome jobs, so the flag was never isolated and its compatibility with -lmm 9 under --loco is unknown. What that control run did find is the real defect: the wrapper does not surface the failing job's error (the batch's stderr goes to a temporary directory that is removed), which hides every GEMMA failure, not just this one.
7 daysdocs: how far the GEMMA pve route gotPjotr
Tried it: passing -pve through gemma-wrapper after -- crashes the wrapper (Ruby backtrace, no output), and the logs the wrapper keeps are the kinship step's -- 223 .log.txt in the cache dir, none containing a pve block, because the per-chromosome LMM logs live in the wrapper's temp dirs and are not retained. Two routes remain, both written down: run GEMMA ourselves (-gk for a text K matrix, then -lmm 1 -pve on one marker, seconds each) and parse 'pve estimate in the null model'; or get -pve into the wrapper's allowlist upstream. The first needs the kinship as plain text, which the wrapper only caches as JSON.
7 daysdocs: why the two engines' h2 disagree, and the three measurements that ↵Pjotr
settle it Written down from a real analysis (BXD.8, 35 phenotyped animals): trait 16769 has both engines saying kinship dominates (BulkLMM h2 = 1.000, sigma2_e = 0; GEMMA clamps l_mle on every marker but the chr2 peak), while 16627 is a genuine disagreement (h2 = 0.000 against a clamped l_mle everywhere). Both sit on a boundary, which is expected at n = 35 -- but opposite boundaries mean the engines are estimating different things: * different kinship matrices (GEMMA's own relatedness from the BIMBAM dosages, one per chromosome under LOCO, against calcKinship's 2*(X-0.5)(X-0.5)'/p + 0.5 with a unit diagonal); * different encodings (0/0.5/1 with NA, against the same dosages with missing values mean-imputed, against R/qtl's two-of-three codes) -- and 16627 is nearly determined by the chr2 genotype; * different variance-component step (once under the null against per-chromosome LOCO, clamped differently at a boundary). The three measurements that would settle it, with what is known about each: 1. GEMMA's pve: it prints "pve estimate" when run with -pve, and gemma-wrapper keeps 223 per-chromosome .log.txt files under its cache dir -- none contain pve, because the flag is not passed. One argument on the GWA step plus a parser. 2. BulkLMM's profile over an h2 grid: BulkLMM.profile_LL(y::Matrix, G, covar, K, h2_grid, markerID; reml) is internal (not exported) and returns (ll_list_null, ll_list_alt); it wants a one-column phenotype matrix and a covariate matrix. An attempt to call it from the driver is not landed -- the signature mismatch is the whole of the remaining work. 3. The kinship matrices, or a summary: calcKinship is one line and can be written per panel, GEMMA's is already cached as *.k.json; correlating off-diagonals says whether the engines disagree about relatedness before arguing about variance components. Until those exist, the reading in the analysis holds: BulkLMM's h2 and the thresholds that rest on it are approximate on a boundary, and GEMMA's 16627 p-values come from a kinship-dominated model.
7 daysplot/bulklmm: the 0.05 and 0.01 lines, never 0.10 unless askedPjotr
Checked what the plot actually draws: it has never drawn the alpha 0.10 line -- the plotter matches 0.05 and 0.01 only. What lists all three is the reply's notes ("alpha 0.01: LOD 4.58, alpha 0.05: 3.65, alpha 0.1: 3.50"), which is where the 0.10 came from. Still, the point stands on its own: 0.10 is the loosest bar in the set and drawing it invites reading it as the line to clear. So the defaults are stated and pinned -- 0.05 and 0.01 on the plot, 0.10 only on request (?all_thresholds=true, or an explicit ?thresholds=) -- alongside GEMMA's Bonferroni, which is opt-in for the same reason: a reference line is an annotation, not a result. tests/bulklmm.rs counts the lines: exactly two by default, and no alpha 0.10 label.
7 daysbulklmm: BULKLMM_BLOCKS as a config option, default 4Pjotr
The marker blocks a single-trait scan is split into were the driver's default (Threads.nthreads()) and changed with --threads. They are a separate knob now: BULKLMM_THREADS Julia threads (--threads) default 4 BULKLMM_BLOCKS independent marker blocks, scanned default 4 in parallel (1 = serial scan) Blocks are exact either way -- LOD 9.589938 and the same permutation thresholds at 1 and 4 -- so the knob only buys wall clock. The reply says what ran ("split into 4 independent blocks and scanned on 4 Julia thread(s) (BULKLMM_BLOCKS / BULKLMM_THREADS ...)"), because the answer to "is it multicore?" should be in the response and not in the log.
7 daysgemma: the permutation refusal names both routes, including running GEMMA itselfPjotr
We do not run GEMMA permutations -- each repeats the whole LOCO scan and the gate gives GEMMA one slot -- but "not supported here" is not the same as "not available". The refusal now says both: 1. POST /compute/bulklmm with nperm: 1000 -- an LMM whose thresholds come from rotations that keep the kinship (R/qtl's nperm does not, and is anti-conservative on a structured panel); 2. run GEMMA's own permutation locally. The reply's notes already name the bimbam pair and the phenotype file, so `gemma-wrapper --permutate` or `gemma -lmm 9 --permutate` on those inputs reproduces exactly the null GEMMA would have used -- hours instead of seconds, which is the honest price of that particular null. docs/bulklmm.md gains the table this belongs in: three ways to get a threshold, ordered by how honest the null is for a structured panel (rotation with kinship; GEMMA's own LOCO permutation run locally; R/qtl's nperm as a sanity check only). SKILL.md says the same in two lines. tests/gemma.rs now asserts both halves of the refusal, and the earlier assertion on /compute/rqtl -- which stopped being true when the refusal was rewritten -- is replaced rather than left passing by accident.
7 daysdocs: what BulkLMM models, what its cores do, and how to package itPjotr
docs/bulklmm.md records the three things that are easy to get wrong when a BulkLMM result is quoted: * where it stands against the other engines -- Haley-Knott with no kinship (R/qtl), LOCO LMM whose permutations are refused (GEMMA), LOCO LMM on HMM probabilities (R/qtl2), and this: a global-kinship LMM whose permutation thresholds are rotation-based and therefore cheap -- the only engine here that can say "genome-wide significant at alpha 0.05" against a null that respects relatedness; * what it is not: global kinship rather than LOCO, so its p-values are not GEMMA's; h2_null says how much relatedness it found (near zero here, which is why it reproduces R/qtl's 9.59 where GEMMA differs); one degree of freedom, so multiparent scans stay with /compute/qtl2; and no standard errors in 1.2.2's scan(), so the rows are marker, chr, pos, lod, beta, h2; * the cores, measured: Threads.@threads is in bulkscan.jl and bulkscan_helpers.jl, never in scan.jl, so a single-trait scan is serial by design and the visible parallelism was BLAS (96 threads 10.5 s against 4 threads 7.7 s). What the driver does about it -- exact marker blocks, cached matrix and kinship, capped BLAS, and the breakdown printed into the reply -- with the numbers (20.9 s at one block against 18.3 s at four, same LOD 9.589938). Also the packaging picture, since it is not obvious from the outside: guix-bioinformatics has a working julia-bin and a julia-build-system, but only three of BulkLMM's closure are packaged, so a pure-Guix build is 35-45 recipes; the pragmatic route is julia-bin plus a depot built once from the pinned Manifest. scripts/prepare-julia-depot.sh is that step, written down, and the depot stays small and GPU-free (79 packages, 139 MB) with JULIA_PKG_OFFLINE=true at request time. The plot test learns that both thresholds are drawn, not one.