# BulkLMM: an LMM scan whose thresholds we can actually compute `POST /compute/bulklmm` runs [BulkLMM.jl](https://github.com/senresearch/BulkLMM.jl) (Apache-2.0) through `scripts/bulklmm_scan.jl`. This note records why the engine is here, what it does and does not model, and what its cores are doing — three things that are easy to get wrong when a result is quoted. ## Why it sits beside GEMMA and R/qtl | | model | kinship | permutations | |---|---|---|---| | `/compute/rqtl` | Haley-Knott (single marker) | none | yes, but shuffles phenotypes — the wrong null for related individuals | | `/compute/gemma` | LMM, LOCO | leave-one-chromosome-out | **refused**: each one re-runs the whole LOCO scan | | `/compute/qtl2` | LMM, LOCO, HMM probabilities | LOCO | no | | `/compute/bulklmm` | LMM, **global** kinship | one K for the whole genome | **yes, cheaply**: rotates by K's eigen decomposition | **Three ways to get a threshold, in order of honesty for a structured panel:** | where | null | cost | |---|---|---| | `/compute/bulklmm` with `nperm` | rotations that keep the kinship | seconds | | GEMMA's own `--permutate`, **run locally** | GEMMA's exact LOCO null | one full LOCO scan per permutation: hours | | R/qtl `nperm` on the same values | phenotypes shuffled across individuals — relatedness destroyed, anti-conservative on RI lines and DO collapses | minutes | The GEMMA endpoint does not run permutations: each repeats the whole LOCO scan and the gate gives GEMMA one slot. Its refusal says both alternatives — and names the files it used (the `bimbam/` pair and the phenotype file) so a user who wants GEMMA's own null can reproduce it locally, which is the only way to get *that* null. So BulkLMM is the only engine here that can say *"this peak is genome-wide significant at α = 0.05"* against a null that respects relatedness. That is what `/compute/gemma`'s `permutations_not_supported` refusal now points at. Worked example, BXD trait 16627 on `BXD.8` (35 phenotyped animals, 21 056 markers): ``` peak rsm10000000888 chr2 4.92 Mb LOD 9.59 beta 0.0027 h2_null ≈ 0 thresholds (200 rotations, seed 42): alpha 0.10 → 3.51, 0.05 → 3.81, 0.01 → 4.47 ``` `rsm10000000888` is the trait's published locus and 9.59 its published LOD, so the engine agrees with the record — and the peak clears the α = 0.01 threshold. ## What it is not * **A global kinship matrix, not LOCO.** One K for every marker; GEMMA excludes each chromosome in turn. Different p-values on the same data, and the reply says so. Do not present a BulkLMM p-value as GEMMA's. * **`h2_null` is the honest tell.** It is the heritability estimated under the null (intercept) model, from `y` and `K` alone. Near zero — as here — the fit is effectively a fixed-effects model, which is exactly why it reproduces R/qtl's answer on this trait where GEMMA's LOCO scan differs. * **One degree of freedom.** BulkLMM 1.2.2 tests one coefficient per marker; a multiparent founder-allele model is `/compute/qtl2`'s job (`crosstype: do`). * **No standard errors yet.** `scan()` in 1.2.2 returns `(b, sigma2_e, h2_null, lod, L_perms)`; `SE` is not among them, so the reply's `se` column does not exist and the rows are `marker, chr, pos, lod, beta, h2`. ## When the two engines' h2 disagree, and what settles it Reported from a real analysis (BXD.8, 35 phenotyped animals): | trait | BulkLMM h2 | GEMMA `l_mle` | reading | |---|---|---|---| | 16769 | 1.000, σ²_e = 0 | at the upper clamp on every marker except the chr2 peak | both say kinship dominates; the peak's drop is the marker absorbing the effect | | 16627 | 0.000, σ²_e = 0.402 | at the upper clamp everywhere, peak included | a genuine disagreement | Both estimates sit on a boundary, which is expected at n = 35 — the likelihood is flat there — so neither number is a precise proportion. But *opposite* boundaries on the same phenotype and the same genotypes means the two engines are estimating different things, and it is worth knowing which: 1. **Different kinship matrices.** GEMMA computes its own relatedness from the BIMBAM dosages (and, in LOCO mode, one per chromosome); BulkLMM's `calcKinship` is `2·(X−0.5)(X−0.5)ᵀ/p + 0.5` with a unit diagonal. Two matrices, two variance components — not comparable until their correlation is known. 2. **Different encodings.** GEMMA gets `0 / 0.5 / 1` dosages with `NA` for missing; BulkLMM gets the same dosages with missing values mean-imputed per marker, and R/qtl's `convert2riself` (two of three codes) is a third. Trait 16627 is nearly determined by the chr2 genotype (B above 1.0, D below 0.6), so the residual variance differs a lot between those encodings. 3. **Different variance-component step.** BulkLMM estimates once under the null with `assumption = "null"` (and REML or ML); GEMMA optimises per chromosome under LOCO. On a boundary both optimisers clamp, and they clamp to different conventions. Measured on **BXD::10001, 34 animals**, once both engines' variance components were readable: BulkLMM's global null `h2` is **0.819** (`σ²_e = 5.381`), and GEMMA's LOCO `pve` over the 20 chromosomes runs **0.3125-0.9558**, median **0.8015**, mean 0.7769. So the two agree on the variance split to within a couple of percent -- 0.80 against 0.82 -- which is the answer this comparison was for: the engines were never in disagreement about heritability, only about which chromosome's kinship to hold the rest of the genome out from. Quote GEMMA's `pve` with its range (it is per chromosome), BulkLMM's `h2` as one number (it is global), and do not compare the p-values: `--loco` weights each chromosome differently, so neither set is the other's null. **What would settle it, in order of cost:** * **GEMMA's own variance components — available, one flag away.** Tested on a synthetic BIMBAM trio (40 individuals, 60 markers) with GEMMA 0.98.6: `-lmm 9 -pve` is accepted and writes the block ## pve estimate in the null model = 1.62846e-06 ## se(pve) in the null model = -nan ## vg estimate in the null model = 0 ## ve estimate in the null model = 0 So the flag works with `-lmm 9`; what was missing was the plumbing, and that is now in. `gemma-wrapper` 5eb53fa copies each run's `.log.txt` **beside** the archive (they were only inside the `.tar.xz`, so reading `pve` meant unpacking a multi-megabyte archive), and `/compute/gemma` passes `-pve` on the GWA step and reports the components per chromosome: > GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451, > chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to 0.9549 > over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg 23.3659, ve > 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933, ve 0.8661). every one of the twenty (only the ends shown here), because under `--loco` each belongs to a different kinship matrix and a caller told only the range cannot ask which chromosome is which. The unrounded values also go to the run log as `variance:` lines, so a `?id=` run's `/compute/gemma/status` stream carries them while the run is in flight. (BXD::10001, 34 animals, LOCO, `-lmm 9`.) That is the number the 16627/16769 comparison lacked: GEMMA's own `h2`, per chromosome, objective function named. Where the two engines still disagree, the disagreement is now between two printed variance components rather than between a number and an inference. Two caveats, both measured. GEMMA prints `-nan` for `se(pve)` when the estimate sits on a boundary, and this build reports that as "not available" rather than as a number; a boundary pve is a direction, not a proportion — the same caveat as BulkLMM's `h2`, from the other engine. And the components are on the phenotype's own scale, so a trait measured in thousandths gives variances in thousandths: BXD::16769 (values 0.0012-0.0122, 36 animals) yields `pve 1.0000 ± 0.0016 (vg 3.411e-5, ve 3.411e-10)` — pve is scale-free and still reads 1.0, but the absolute numbers are meaningless to compare across traits. Compare `pve` with `h2`, not `vg` with `σ²_e`. * **BulkLMM's likelihood over an h2 grid.** The package has `BulkLMM.profile_LL(y::Matrix, G, covar, K, h2_grid, markerID; reml)` (internal, not exported), returning `(ll_list_null, ll_list_alt)` over the grid. It wants the phenotype as a one-column matrix and a covariate matrix (an all-zero column is what `scan` uses with no covariates). Printing that profile at the peak marker says whether an h2 of 0.000 is the maximum or the flat edge of one — the difference between "no heritability" and "the estimator cannot tell". * **The kinship matrices, or a summary.** BulkLMM's `calcKinship(g)` is one line and can be written out per panel; GEMMA's is already cached as a `-gk` `*.k.json`. Correlating their off-diagonals (or reporting min/median/max) is enough to say whether the two engines disagree about *relatedness* before arguing about what the variance components mean. Until then, the honest reading is the one the analysis reached: BulkLMM's `h2` and the thresholds that depend on it are approximate where `h2` is on a boundary, and GEMMA's p-values for 16627 come from a kinship-dominated model — its peak should not be compared with a fixed-effects LOD. ## Cores: `scan()` is serial by design This was measured, not assumed. `Threads.@threads` appears in `bulkscan.jl` and `bulkscan_helpers.jl` — **never in `scan.jl`**. The single-trait `scan()` this endpoint calls runs on one Julia thread; the parallel speed-up that *is* visible comes from BLAS: ``` --threads 1 scan 13.7 s BLAS 96 threads scan 10.5 s --threads 4 scan 12.3 s BLAS 4 threads scan 7.7 s (better!) ``` OpenBLAS defaults to every core on the machine (96 here) and oversubscribes on matrices of this shape. `scripts/bulklmm_scan.jl` therefore: * **splits the markers into independent blocks** and scans them on Julia threads (`--blocks`, `BULKLMM_BLOCKS`, default 4; `1` is a plain serial scan). It is *exact*: each marker's LOD is independent once the null variance components are fixed, and those depend on `y` and `K` only. Verified identical output — LOD 9.589938 and the same thresholds at 1 and 4 blocks — and the permutation thresholds are combined by taking each permutation's maximum across blocks before the quantiles, which is precisely what a genome-wide threshold is; * **caches the parsed matrix and the kinship** per panel (`read 1.3 → 0.4 s`, kinship `0.9 → 0`); * **caps BLAS** at 4 threads when serial, 1 per block when parallel; * draws two reference lines: the genome-wide alpha 0.05 and the stricter alpha 0.01. The reply reports alpha 0.10 as well, but that is the loosest bar in the set, so it is drawn only on request (`?all_thresholds=true`, or an explicit `?thresholds=`), and GEMMA's Bonferroni line on `/plot/gemma` is opt-in in the same spirit; * reports what ran in the reply: `julia threads 4, blocks 4, BLAS threads 96 | read 1.5 s, scan 10.5 s`. What that buys at this size: 20.9 s (1 block) → 18.3 s (4 blocks). Modest, because the LMM loop is ~8 s of the run, each block re-estimates its variance components, and Julia's startup is the rest. **Where cores pay is `bulkscan()`** — many traits in one process, threaded over trait blocks — which is the shape an eQTL endpoint wants and the next thing to add here. ## Configuration and the Julia side | key | meaning | |---|---| | `BULKLMM_CMD` | the `julia` binary (default: the store's `julia-1.8.5`) | | `BULKLMM_ENV` | the project its packages are instantiated into | | `BULKLMM_DEPOT` | the depot those packages live in (default `$HOME/.julia`) | | `BULKLMM_SCRIPT` | `scripts/bulklmm_scan.jl` | | `BULKLMM_THREADS` | Julia threads (`--threads`), default **4** | | `BULKLMM_BLOCKS` | independent marker blocks scanned in parallel, default **4** (`1` = serial scan, same answer) | | `BULKLMM_CACHE_DIR` | scratch: prepared genotypes, cached matrix and kinship | | `BULKLMM_PLOTTER` | `scripts/plot-bulklmm.py` for `POST /plot/bulklmm` | The depot is small and deliberately so — **79 packages, 139 MB, no GPU/CUDA stack**: BulkLMM's closure is CSV, DataFrames, Distributions, Optim and their dependencies. Every request runs with `JULIA_PKG_OFFLINE=true`, so a scan can never fetch a package; if something is missing, the run fails saying which package, and the fix is to prepare the depot — a deliberate operator action, not a side effect of a request. **Packaging, current state.** The deployment uses the store's own `julia-1.8.5` with a depot under `~/.julia`, and the project (`BULKLMM_ENV`) pinned by `Manifest.toml` (BulkLMM 1.2.2 + CSV + DataFrames). `guix-bioinformatics` already carries a working `julia-bin` (1.10.9) and a `julia-build-system`, but of BulkLMM's closure only `julia-distributions`, `julia-optim` and `julia-nlsolversbase` are packaged there — a pure-Guix build would need ~35–45 new package recipes. The pragmatic route, if the depot should stop being `~/.julia`, is `julia-bin` plus a depot built once from our `Manifest.toml` (`scripts/prepare-julia-depot.sh`) and pointed at by `BULKLMM_DEPOT`; the alternative, `PackageCompiler`, would bake the driver into a single executable and remove Julia startup from the request path entirely. ## Traps worth remembering * `--project=`, **not** `--project `: Julia 1.8 takes the space-separated form as a program argument and the run produces nothing, with exit status 0. * A `NaN` LOD parses as a valid `f64` in Rust; the parser filters non-finite values, or a marker the model could not fit would be reported as a LOD of 0. * The gate gives a caller one BulkLMM scan at a time (`compute_busy`), which is why two tests that scan concurrently must tolerate a 409 rather than fail. ## Deployed state (2026-11-11) Both processes were restarted with `GEMMA_WRAPPER_CMD` pointing at `gemma-wrapper-1.00-pre1-4.5eb53fa` (the package is pinned to `5eb53fa`, revision 4, in `guix-bioinformatics`' `gn/packages/gemma.scm`), because the profile's own wrapper is older than the pin. Verified against the running production instance rather than a fixture: `POST /compute/gemma` for BXD::10001 returns `pve 0.3125 to 0.9558 over 20 chromosomes` in its notes.