| Age | Commit message (Collapse) | Author |
|
A failing scan came back as
BulkLMM failed: in expression starting at scripts/bulklmm_scan.jl:26
which names neither the package nor the reason. Julia had said it plainly,
two lines up:
ERROR: LoadError: ArgumentError: Package CSV not found in current path.
- Run `import Pkg; Pkg.add("CSV")` to install the CSV package.
The complaint finder looked for a line starting "Error" -- Julia writes
"ERROR:" -- and fell back to the last non-empty line, which is the line
number. That is the same defect as gemma-wrapper's Ruby backtrace, in our
own error handling: the wrapped tool said why, and we threw it away.
julia_complaint takes the first ERROR:/LoadError line plus Julia's hint
line when there is one, with the last line kept only as a fallback for a
driver that fails silently. Three unit tests cover the three shapes.
The integration tests also stop failing on a half-instantiated Julia
project: bulklmm_available used to accept any directory with a
Project.toml, and /tmp/gj-env has one naming BulkLMM and nothing else, so
the scan died on `using CSV` instead of skipping. It now requires the
project to name the three packages the driver loads, which is what a
usable env is.
|
|
Checked what the plot actually draws: it has never drawn the alpha 0.10
line -- the plotter matches 0.05 and 0.01 only. What lists all three is the
reply's notes ("alpha 0.01: LOD 4.58, alpha 0.05: 3.65, alpha 0.1: 3.50"),
which is where the 0.10 came from.
Still, the point stands on its own: 0.10 is the loosest bar in the set and
drawing it invites reading it as the line to clear. So the defaults are
stated and pinned -- 0.05 and 0.01 on the plot, 0.10 only on request
(?all_thresholds=true, or an explicit ?thresholds=) -- alongside GEMMA's
Bonferroni, which is opt-in for the same reason: a reference line is an
annotation, not a result.
tests/bulklmm.rs counts the lines: exactly two by default, and no
alpha 0.10 label.
|
|
The marker blocks a single-trait scan is split into were the driver's
default (Threads.nthreads()) and changed with --threads. They are a
separate knob now:
BULKLMM_THREADS Julia threads (--threads) default 4
BULKLMM_BLOCKS independent marker blocks, scanned default 4
in parallel (1 = serial scan)
Blocks are exact either way -- LOD 9.589938 and the same permutation
thresholds at 1 and 4 -- so the knob only buys wall clock. The reply says
what ran ("split into 4 independent blocks and scanned on 4 Julia
thread(s) (BULKLMM_BLOCKS / BULKLMM_THREADS ...)"), because the answer to
"is it multicore?" should be in the response and not in the log.
|
|
We do not run GEMMA permutations -- each repeats the whole LOCO scan and the
gate gives GEMMA one slot -- but "not supported here" is not the same as
"not available". The refusal now says both:
1. POST /compute/bulklmm with nperm: 1000 -- an LMM whose thresholds come
from rotations that keep the kinship (R/qtl's nperm does not, and is
anti-conservative on a structured panel);
2. run GEMMA's own permutation locally. The reply's notes already name the
bimbam pair and the phenotype file, so `gemma-wrapper --permutate` or
`gemma -lmm 9 --permutate` on those inputs reproduces exactly the null
GEMMA would have used -- hours instead of seconds, which is the honest
price of that particular null.
docs/bulklmm.md gains the table this belongs in: three ways to get a
threshold, ordered by how honest the null is for a structured panel
(rotation with kinship; GEMMA's own LOCO permutation run locally; R/qtl's
nperm as a sanity check only). SKILL.md says the same in two lines.
tests/gemma.rs now asserts both halves of the refusal, and the earlier
assertion on /compute/rqtl -- which stopped being true when the refusal was
rewritten -- is replaced rather than left passing by accident.
|
|
docs/bulklmm.md records the three things that are easy to get wrong when a
BulkLMM result is quoted:
* where it stands against the other engines -- Haley-Knott with no
kinship (R/qtl), LOCO LMM whose permutations are refused (GEMMA),
LOCO LMM on HMM probabilities (R/qtl2), and this: a global-kinship
LMM whose permutation thresholds are rotation-based and therefore
cheap -- the only engine here that can say "genome-wide significant
at alpha 0.05" against a null that respects relatedness;
* what it is not: global kinship rather than LOCO, so its p-values are
not GEMMA's; h2_null says how much relatedness it found (near zero
here, which is why it reproduces R/qtl's 9.59 where GEMMA differs);
one degree of freedom, so multiparent scans stay with /compute/qtl2;
and no standard errors in 1.2.2's scan(), so the rows are
marker, chr, pos, lod, beta, h2;
* the cores, measured: Threads.@threads is in bulkscan.jl and
bulkscan_helpers.jl, never in scan.jl, so a single-trait scan is
serial by design and the visible parallelism was BLAS (96 threads
10.5 s against 4 threads 7.7 s). What the driver does about it --
exact marker blocks, cached matrix and kinship, capped BLAS, and the
breakdown printed into the reply -- with the numbers (20.9 s at one
block against 18.3 s at four, same LOD 9.589938).
Also the packaging picture, since it is not obvious from the outside:
guix-bioinformatics has a working julia-bin and a julia-build-system, but
only three of BulkLMM's closure are packaged, so a pure-Guix build is
35-45 recipes; the pragmatic route is julia-bin plus a depot built once
from the pinned Manifest. scripts/prepare-julia-depot.sh is that step,
written down, and the depot stays small and GPU-free (79 packages,
139 MB) with JULIA_PKG_OFFLINE=true at request time.
The plot test learns that both thresholds are drawn, not one.
|
|
Reported: a BulkLMM run does not look multicore. It is not a
misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint
calls -- has no `Threads.@threads` at all; the package threads
`bulkscan()` (many traits) and bulkscan_helpers, not scan. The
parallelism that was visible before was BLAS (OpenBLAS defaulting to all
96 cores here), which is why `user` time ran 4x `real` while Julia
threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s.
* The driver splits the markers into independent blocks and scans them
on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact:
the LOD for each marker is independent given the null variance
components, which depend on y and K only. Verified identical -- peak
LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks --
and permutation thresholds combine by taking each permutation's
maximum across blocks before the quantiles, which is what a
genome-wide threshold is.
* The parsed matrix and the kinship are cached per panel (read 1.3 s ->
0.4 s, kinship 0.9 s -> 0), because both depend only on the cross.
* The summary in `notes` now says what ran: julia threads, blocks, BLAS
threads, and the read/scan split.
* The hand parser is back: CSV.jl into a DataFrame is an order of
magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s).
* BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for
one scan).
Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks
18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of
the run and each block re-estimates its variance components. Where the
cores pay is bulkscan() -- many traits in one process, threaded over
trait blocks -- which is the shape the next endpoint wants; documented in
the README rather than left as folklore.
|
|
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values
have no threshold we can compute: a GEMMA permutation repeats the whole
scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual
stand-in and it is the wrong null -- it shuffles phenotypes as if the
individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global
kinship matrix and permutes by rotating the data through the kinship's
eigen decomposition, so a threshold costs seconds.
POST /compute/bulklmm {group, genotype_version, samples|basket,
nperm, seed, reml, assumption}
Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8,
35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2
4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81,
0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's
published locus and LOD, and a significant peak against a threshold this
API could not compute before.
Two differences from GEMMA are in every reply, not implied: a global
kinship matrix rather than leave-one-chromosome-out, and h2_null saying
how much relatedness was found (here ~0, so the fit is effectively
fixed-effects -- which is why it agrees with R/qtl where GEMMA differs).
* src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a
strains x markers matrix (our BIMBAM pair is already the allele
dosages BulkLMM wants), a marker map, the phenotype and covariates
-- runs the driver through streaming::run_streaming_with_env, and
reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid
f64 in Rust and would otherwise be reported as a value.
* scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds),
with the seed recorded so a threshold set is reproducible.
* gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is
cheap, unlike R/qtl's.
* Julia comes from the deployment's own store binary with a prepared
depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request
runs JULIA_PKG_OFFLINE=true, so a scan can never download anything.
`--project=<path>`, not `--project <path>`: Julia 1.8 takes the
space-separated form as a program argument.
tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and
seed are in the notes, and a scan without permutations says there is no
threshold rather than leaving a bare LOD to be read as significant.
|