BulkLMM: an LMM scan whose thresholds we can actually compute
POST /compute/bulklmm runs BulkLMM.jl
(Apache-2.0) through scripts/bulklmm_scan.jl. This note records why the engine
is here, what it does and does not model, and what its cores are doing — three
things that are easy to get wrong when a result is quoted.
Why it sits beside GEMMA and R/qtl
| model | kinship | permutations | |
|---|---|---|---|
/compute/rqtl |
Haley-Knott (single marker) | none | yes, but shuffles phenotypes — the wrong null for related individuals |
/compute/gemma |
LMM, LOCO | leave-one-chromosome-out | refused: each one re-runs the whole LOCO scan |
/compute/qtl2 |
LMM, LOCO, HMM probabilities | LOCO | no |
/compute/bulklmm |
LMM, global kinship | one K for the whole genome | yes, cheaply: rotates by K's eigen decomposition |
Three ways to get a threshold, in order of honesty for a structured panel:
| where | null | cost |
|---|---|---|
/compute/bulklmm with nperm |
rotations that keep the kinship | seconds |
GEMMA's own --permutate, run locally |
GEMMA's exact LOCO null | one full LOCO scan per permutation: hours |
R/qtl nperm on the same values |
phenotypes shuffled across individuals — relatedness destroyed, anti-conservative on RI lines and DO collapses | minutes |
The GEMMA endpoint does not run permutations: each repeats the whole LOCO scan
and the gate gives GEMMA one slot. Its refusal says both alternatives — and
names the files it used (the bimbam/ pair and the phenotype file) so a user who
wants GEMMA's own null can reproduce it locally, which is the only way to get
that null.
So BulkLMM is the only engine here that can say "this peak is genome-wide
significant at α = 0.05" against a null that respects relatedness. That is what
/compute/gemma's permutations_not_supported refusal now points at.
Worked example, BXD trait 16627 on BXD.8 (35 phenotyped animals, 21 056
markers):
peak rsm10000000888 chr2 4.92 Mb LOD 9.59 beta 0.0027 h2_null ≈ 0
thresholds (200 rotations, seed 42): alpha 0.10 → 3.51, 0.05 → 3.81, 0.01 → 4.47
rsm10000000888 is the trait's published locus and 9.59 its published LOD, so
the engine agrees with the record — and the peak clears the α = 0.01 threshold.
What it is not
- A global kinship matrix, not LOCO. One K for every marker; GEMMA excludes each chromosome in turn. Different p-values on the same data, and the reply says so. Do not present a BulkLMM p-value as GEMMA's.
h2_nullis the honest tell. It is the heritability estimated under the null (intercept) model, fromyandKalone. Near zero — as here — the fit is effectively a fixed-effects model, which is exactly why it reproduces R/qtl's answer on this trait where GEMMA's LOCO scan differs.- One degree of freedom. BulkLMM 1.2.2 tests one coefficient per marker; a
multiparent founder-allele model is
/compute/qtl2's job (crosstype: do). - No standard errors yet.
scan()in 1.2.2 returns(b, sigma2_e, h2_null, lod, L_perms);SEis not among them, so the reply'ssecolumn does not exist and the rows aremarker, chr, pos, lod, beta, h2.
When the two engines' h2 disagree, and what settles it
Reported from a real analysis (BXD.8, 35 phenotyped animals):
| trait | BulkLMM h2 | GEMMA l_mle |
reading |
|---|---|---|---|
| 16769 | 1.000, σ²_e = 0 | at the upper clamp on every marker except the chr2 peak | both say kinship dominates; the peak's drop is the marker absorbing the effect |
| 16627 | 0.000, σ²_e = 0.402 | at the upper clamp everywhere, peak included | a genuine disagreement |
Both estimates sit on a boundary, which is expected at n = 35 — the likelihood is flat there — so neither number is a precise proportion. But opposite boundaries on the same phenotype and the same genotypes means the two engines are estimating different things, and it is worth knowing which:
- Different kinship matrices. GEMMA computes its own relatedness from the
BIMBAM dosages (and, in LOCO mode, one per chromosome); BulkLMM's
calcKinshipis2·(X−0.5)(X−0.5)ᵀ/p + 0.5with a unit diagonal. Two matrices, two variance components — not comparable until their correlation is known. - Different encodings. GEMMA gets
0 / 0.5 / 1dosages withNAfor missing; BulkLMM gets the same dosages with missing values mean-imputed per marker, and R/qtl'sconvert2riself(two of three codes) is a third. Trait 16627 is nearly determined by the chr2 genotype (B above 1.0, D below 0.6), so the residual variance differs a lot between those encodings. - Different variance-component step. BulkLMM estimates once under the null
with
assumption = "null"(and REML or ML); GEMMA optimises per chromosome under LOCO. On a boundary both optimisers clamp, and they clamp to different conventions.
Measured on BXD::10001, 34 animals, once both engines' variance
components were readable: BulkLMM's global null h2 is 0.819
(σ²_e = 5.381), and GEMMA's LOCO pve over the 20 chromosomes runs
0.3125-0.9558, median 0.8015, mean 0.7769. So the two agree on the
variance split to within a couple of percent -- 0.80 against 0.82 -- which
is the answer this comparison was for: the engines were never in
disagreement about heritability, only about which chromosome's kinship to
hold the rest of the genome out from. Quote GEMMA's pve with its range
(it is per chromosome), BulkLMM's h2 as one number (it is global), and do
not compare the p-values: --loco weights each chromosome differently, so
neither set is the other's null.
What would settle it, in order of cost:
-
GEMMA's own variance components — available, one flag away. Tested on a synthetic BIMBAM trio (40 individuals, 60 markers) with GEMMA 0.98.6:
-lmm 9 -pveis accepted and writes the block## pve estimate in the null model = 1.62846e-06 ## se(pve) in the null model = -nan ## vg estimate in the null model = 0 ## ve estimate in the null model = 0
So the flag works with -lmm 9; what was missing was the plumbing, and that
is now in. gemma-wrapper 5eb53fa copies each run's .log.txt beside the
archive (they were only inside the .tar.xz, so reading pve meant unpacking
a multi-megabyte archive), and /compute/gemma passes -pve on the GWA step
and reports the components per chromosome:
GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451, chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to 0.9549 over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg 23.3659, ve 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933, ve 0.8661).
every one of the twenty (only the ends shown here), because under --loco
each belongs to a different kinship matrix and a caller told only the range
cannot ask which chromosome is which. The unrounded values also go to the run
log as variance: lines, so a ?id= run's
/compute/gemma/status stream carries them while the run is in flight.
(BXD::10001, 34 animals, LOCO, -lmm 9.) That is the number the 16627/16769
comparison lacked: GEMMA's own h2, per chromosome, objective function
named. Where the two engines still disagree, the disagreement is now between
two printed variance components rather than between a number and an
inference.
Two caveats, both measured. GEMMA prints -nan for se(pve) when the
estimate sits on a boundary, and this build reports that as "not available"
rather than as a number; a boundary pve is a direction, not a proportion — the
same caveat as BulkLMM's h2, from the other engine. And the components are
on the phenotype's own scale, so a trait measured in thousandths gives
variances in thousandths: BXD::16769 (values 0.0012-0.0122, 36 animals) yields
pve 1.0000 ± 0.0016 (vg 3.411e-5, ve 3.411e-10) — pve is scale-free and
still reads 1.0, but the absolute numbers are meaningless to compare across
traits. Compare pve with h2, not vg with σ²_e.
* BulkLMM's likelihood over an h2 grid. The package has
BulkLMM.profile_LL(y::Matrix, G, covar, K, h2_grid, markerID; reml) (internal,
not exported), returning (ll_list_null, ll_list_alt) over the grid. It wants
the phenotype as a one-column matrix and a covariate matrix (an all-zero column
is what scan uses with no covariates). Printing that profile at the peak
marker says whether an h2 of 0.000 is the maximum or the flat edge of one — the
difference between "no heritability" and "the estimator cannot tell".
* The kinship matrices, or a summary. BulkLMM's calcKinship(g) is one line
and can be written out per panel; GEMMA's is already cached as a -gk
*.k.json. Correlating their off-diagonals (or reporting min/median/max) is
enough to say whether the two engines disagree about relatedness before
arguing about what the variance components mean.
Until then, the honest reading is the one the analysis reached: BulkLMM's h2
and the thresholds that depend on it are approximate where h2 is on a
boundary, and GEMMA's p-values for 16627 come from a kinship-dominated model —
its peak should not be compared with a fixed-effects LOD.
Cores: scan() is serial by design
This was measured, not assumed. Threads.@threads appears in bulkscan.jl and
bulkscan_helpers.jl — never in scan.jl. The single-trait scan() this
endpoint calls runs on one Julia thread; the parallel speed-up that is visible
comes from BLAS:
--threads 1 scan 13.7 s BLAS 96 threads scan 10.5 s
--threads 4 scan 12.3 s BLAS 4 threads scan 7.7 s (better!)
OpenBLAS defaults to every core on the machine (96 here) and oversubscribes on
matrices of this shape. scripts/bulklmm_scan.jl therefore:
- splits the markers into independent blocks and scans them on Julia
threads (
--blocks,BULKLMM_BLOCKS, default 4;1is a plain serial scan). It is exact: each marker's LOD is independent once the null variance components are fixed, and those depend onyandKonly. Verified identical output — LOD 9.589938 and the same thresholds at 1 and 4 blocks — and the permutation thresholds are combined by taking each permutation's maximum across blocks before the quantiles, which is precisely what a genome-wide threshold is; - caches the parsed matrix and the kinship per panel (
read 1.3 → 0.4 s, kinship0.9 → 0); - caps BLAS at 4 threads when serial, 1 per block when parallel;
- draws two reference lines: the genome-wide alpha 0.05 and the stricter
alpha 0.01. The reply reports alpha 0.10 as well, but that is the loosest
bar in the set, so it is drawn only on request (
?all_thresholds=true, or an explicit?thresholds=), and GEMMA's Bonferroni line on/plot/gemmais opt-in in the same spirit; - reports what ran in the reply:
julia threads 4, blocks 4, BLAS threads 96 | read 1.5 s, scan 10.5 s.
What that buys at this size: 20.9 s (1 block) → 18.3 s (4 blocks). Modest,
because the LMM loop is ~8 s of the run, each block re-estimates its variance
components, and Julia's startup is the rest. Where cores pay is
bulkscan() — many traits in one process, threaded over trait blocks — which
is the shape an eQTL endpoint wants and the next thing to add here.
Configuration and the Julia side
| key | meaning |
|---|---|
BULKLMM_CMD |
the julia binary (default: the store's julia-1.8.5) |
BULKLMM_ENV |
the project its packages are instantiated into |
BULKLMM_DEPOT |
the depot those packages live in (default $HOME/.julia) |
BULKLMM_SCRIPT |
scripts/bulklmm_scan.jl |
BULKLMM_THREADS |
Julia threads (--threads), default 4 |
BULKLMM_BLOCKS |
independent marker blocks scanned in parallel, default 4 (1 = serial scan, same answer) |
BULKLMM_CACHE_DIR |
scratch: prepared genotypes, cached matrix and kinship |
BULKLMM_PLOTTER |
scripts/plot-bulklmm.py for POST /plot/bulklmm |
The depot is small and deliberately so — 79 packages, 139 MB, no GPU/CUDA
stack: BulkLMM's closure is CSV, DataFrames, Distributions, Optim and their
dependencies. Every request runs with JULIA_PKG_OFFLINE=true, so a scan can
never fetch a package; if something is missing, the run fails saying which
package, and the fix is to prepare the depot — a deliberate operator action,
not a side effect of a request.
Packaging, current state. The deployment uses the store's own
julia-1.8.5 with a depot under ~/.julia, and the project (BULKLMM_ENV)
pinned by Manifest.toml (BulkLMM 1.2.2 + CSV + DataFrames). guix-bioinformatics
already carries a working julia-bin (1.10.9) and a julia-build-system, but
of BulkLMM's closure only julia-distributions, julia-optim and
julia-nlsolversbase are packaged there — a pure-Guix build would need ~35–45
new package recipes. The pragmatic route, if the depot should stop being
~/.julia, is julia-bin plus a depot built once from our Manifest.toml
(scripts/prepare-julia-depot.sh) and pointed at by BULKLMM_DEPOT; the
alternative, PackageCompiler, would bake the driver into a single executable
and remove Julia startup from the request path entirely.
Traps worth remembering
--project=<path>, not--project <path>: Julia 1.8 takes the space-separated form as a program argument and the run produces nothing, with exit status 0.- A
NaNLOD parses as a validf64in Rust; the parser filters non-finite values, or a marker the model could not fit would be reported as a LOD of 0. - The gate gives a caller one BulkLMM scan at a time (
compute_busy), which is why two tests that scan concurrently must tolerate a 409 rather than fail.
Deployed state (2026-11-11)
Both processes were restarted with GEMMA_WRAPPER_CMD pointing at
gemma-wrapper-1.00-pre1-4.5eb53fa (the package is pinned to 5eb53fa,
revision 4, in guix-bioinformatics' gn/packages/gemma.scm), because the
profile's own wrapper is older than the pin. Verified against the running
production instance rather than a fixture: POST /compute/gemma for
BXD::10001 returns pve 0.3125 to 0.9558 over 20 chromosomes in its notes.
