about summary refs log tree commit diff
path: root/src/bulklmm.rs
AgeCommit message (Collapse)Author
7 daysbulklmm: report Julia's error line, not its last linePjotr
A failing scan came back as BulkLMM failed: in expression starting at scripts/bulklmm_scan.jl:26 which names neither the package nor the reason. Julia had said it plainly, two lines up: ERROR: LoadError: ArgumentError: Package CSV not found in current path. - Run `import Pkg; Pkg.add("CSV")` to install the CSV package. The complaint finder looked for a line starting "Error" -- Julia writes "ERROR:" -- and fell back to the last non-empty line, which is the line number. That is the same defect as gemma-wrapper's Ruby backtrace, in our own error handling: the wrapped tool said why, and we threw it away. julia_complaint takes the first ERROR:/LoadError line plus Julia's hint line when there is one, with the last line kept only as a fallback for a driver that fails silently. Three unit tests cover the three shapes. The integration tests also stop failing on a half-instantiated Julia project: bulklmm_available used to accept any directory with a Project.toml, and /tmp/gj-env has one naming BulkLMM and nothing else, so the scan died on `using CSV` instead of skipping. It now requires the project to name the three packages the driver loads, which is what a usable env is.
7 daysbulklmm: report the variance components, and say what a boundary meansPjotr
"is the kinship term doing anything?" should be answerable from the reply. The driver now prints h2 and the residual variance under the null model, and the reply carries both: Variance components: h2 = 0.000000, sigma2_e = 0.402171, estimated once under the null (intercept) model from y and K, not per marker. With a few dozen phenotyped individuals these estimates pile up at the boundaries 0 and 1 -- the likelihood is flat there -- so read h2 as "kinship matters a lot / not at all" rather than as a precise proportion. At a boundary (h2 ~ 0 the model is ordinary regression, which is why its LODs can match R/qtl's to the digit). Checked with a strain-ID-driven trait (values = rank of strain), which has h2 ~ 1 by construction: BulkLMM reports h2 = 1.000000, sigma2_e = 3e-06 and a peak of LOD 3.1 -- the kinship term absorbs the strain structure -- while R/qtl, which has no kinship term at all, reports a spurious peak of 5.06 on the same values. That is the discrimination the h2 column was missing.
7 daysbulklmm: BULKLMM_BLOCKS as a config option, default 4Pjotr
The marker blocks a single-trait scan is split into were the driver's default (Threads.nthreads()) and changed with --threads. They are a separate knob now: BULKLMM_THREADS Julia threads (--threads) default 4 BULKLMM_BLOCKS independent marker blocks, scanned default 4 in parallel (1 = serial scan) Blocks are exact either way -- LOD 9.589938 and the same permutation thresholds at 1 and 4 -- so the knob only buys wall clock. The reply says what ran ("split into 4 independent blocks and scanned on 4 Julia thread(s) (BULKLMM_BLOCKS / BULKLMM_THREADS ...)"), because the answer to "is it multicore?" should be in the response and not in the log.
7 daysbulklmm: put the whole driver summary in the replyPjotr
The threads/blocks/read/scan breakdown and the permutation thresholds are two separate lines on stderr, and the reply only carried the last one -- so the answer to "is it multicore?" was in the log rather than in the response.
8 daysbulklmm: an LMM scan that can be asked for thresholdsPjotr
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values have no threshold we can compute: a GEMMA permutation repeats the whole scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual stand-in and it is the wrong null -- it shuffles phenotypes as if the individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global kinship matrix and permutes by rotating the data through the kinship's eigen decomposition, so a threshold costs seconds. POST /compute/bulklmm {group, genotype_version, samples|basket, nperm, seed, reml, assumption} Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8, 35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2 4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81, 0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's published locus and LOD, and a significant peak against a threshold this API could not compute before. Two differences from GEMMA are in every reply, not implied: a global kinship matrix rather than leave-one-chromosome-out, and h2_null saying how much relatedness was found (here ~0, so the fit is effectively fixed-effects -- which is why it agrees with R/qtl where GEMMA differs). * src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a strains x markers matrix (our BIMBAM pair is already the allele dosages BulkLMM wants), a marker map, the phenotype and covariates -- runs the driver through streaming::run_streaming_with_env, and reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid f64 in Rust and would otherwise be reported as a value. * scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds), with the seed recorded so a threshold set is reproducible. * gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is cheap, unlike R/qtl's. * Julia comes from the deployment's own store binary with a prepared depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request runs JULIA_PKG_OFFLINE=true, so a scan can never download anything. `--project=<path>`, not `--project <path>`: Julia 1.8 takes the space-separated form as a program argument. tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and seed are in the notes, and a scan without permutations says there is no threshold rather than leaving a bare LOD to be read as significant.