1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
|
# BulkLMM: an LMM scan whose thresholds we can actually compute
`POST /compute/bulklmm` runs [BulkLMM.jl](https://github.com/senresearch/BulkLMM.jl)
(Apache-2.0) through `scripts/bulklmm_scan.jl`. This note records why the engine
is here, what it does and does not model, and what its cores are doing — three
things that are easy to get wrong when a result is quoted.
## Why it sits beside GEMMA and R/qtl
| | model | kinship | permutations |
|---|---|---|---|
| `/compute/rqtl` | Haley-Knott (single marker) | none | yes, but shuffles phenotypes — the wrong null for related individuals |
| `/compute/gemma` | LMM, LOCO | leave-one-chromosome-out | **refused**: each one re-runs the whole LOCO scan |
| `/compute/qtl2` | LMM, LOCO, HMM probabilities | LOCO | no |
| `/compute/bulklmm` | LMM, **global** kinship | one K for the whole genome | **yes, cheaply**: rotates by K's eigen decomposition |
**Three ways to get a threshold, in order of honesty for a structured panel:**
| where | null | cost |
|---|---|---|
| `/compute/bulklmm` with `nperm` | rotations that keep the kinship | seconds |
| GEMMA's own `--permutate`, **run locally** | GEMMA's exact LOCO null | one full LOCO scan per permutation: hours |
| R/qtl `nperm` on the same values | phenotypes shuffled across individuals — relatedness destroyed, anti-conservative on RI lines and DO collapses | minutes |
The GEMMA endpoint does not run permutations: each repeats the whole LOCO scan
and the gate gives GEMMA one slot. Its refusal says both alternatives — and
names the files it used (the `bimbam/` pair and the phenotype file) so a user who
wants GEMMA's own null can reproduce it locally, which is the only way to get
*that* null.
So BulkLMM is the only engine here that can say *"this peak is genome-wide
significant at α = 0.05"* against a null that respects relatedness. That is what
`/compute/gemma`'s `permutations_not_supported` refusal now points at.
Worked example, BXD trait 16627 on `BXD.8` (35 phenotyped animals, 21 056
markers):
```
peak rsm10000000888 chr2 4.92 Mb LOD 9.59 beta 0.0027 h2_null ≈ 0
thresholds (200 rotations, seed 42): alpha 0.10 → 3.51, 0.05 → 3.81, 0.01 → 4.47
```
`rsm10000000888` is the trait's published locus and 9.59 its published LOD, so
the engine agrees with the record — and the peak clears the α = 0.01 threshold.
## What it is not
* **A global kinship matrix, not LOCO.** One K for every marker; GEMMA excludes
each chromosome in turn. Different p-values on the same data, and the reply
says so. Do not present a BulkLMM p-value as GEMMA's.
* **`h2_null` is the honest tell.** It is the heritability estimated under the
null (intercept) model, from `y` and `K` alone. Near zero — as here — the fit
is effectively a fixed-effects model, which is exactly why it reproduces
R/qtl's answer on this trait where GEMMA's LOCO scan differs.
* **One degree of freedom.** BulkLMM 1.2.2 tests one coefficient per marker; a
multiparent founder-allele model is `/compute/qtl2`'s job (`crosstype: do`).
* **No standard errors yet.** `scan()` in 1.2.2 returns `(b, sigma2_e, h2_null,
lod, L_perms)`; `SE` is not among them, so the reply's `se` column does not
exist and the rows are `marker, chr, pos, lod, beta, h2`.
## When the two engines' h2 disagree, and what settles it
Reported from a real analysis (BXD.8, 35 phenotyped animals):
| trait | BulkLMM h2 | GEMMA `l_mle` | reading |
|---|---|---|---|
| 16769 | 1.000, σ²_e = 0 | at the upper clamp on every marker except the chr2 peak | both say kinship dominates; the peak's drop is the marker absorbing the effect |
| 16627 | 0.000, σ²_e = 0.402 | at the upper clamp everywhere, peak included | a genuine disagreement |
Both estimates sit on a boundary, which is expected at n = 35 — the likelihood
is flat there — so neither number is a precise proportion. But *opposite*
boundaries on the same phenotype and the same genotypes means the two engines
are estimating different things, and it is worth knowing which:
1. **Different kinship matrices.** GEMMA computes its own relatedness from the
BIMBAM dosages (and, in LOCO mode, one per chromosome); BulkLMM's
`calcKinship` is `2·(X−0.5)(X−0.5)ᵀ/p + 0.5` with a unit diagonal. Two
matrices, two variance components — not comparable until their correlation is
known.
2. **Different encodings.** GEMMA gets `0 / 0.5 / 1` dosages with `NA` for
missing; BulkLMM gets the same dosages with missing values mean-imputed per
marker, and R/qtl's `convert2riself` (two of three codes) is a third.
Trait 16627 is nearly determined by the chr2 genotype (B above 1.0, D below
0.6), so the residual variance differs a lot between those encodings.
3. **Different variance-component step.** BulkLMM estimates once under the null
with `assumption = "null"` (and REML or ML); GEMMA optimises per chromosome
under LOCO. On a boundary both optimisers clamp, and they clamp to different
conventions.
Measured on **BXD::10001, 34 animals**, once both engines' variance
components were readable: BulkLMM's global null `h2` is **0.819**
(`σ²_e = 5.381`), and GEMMA's LOCO `pve` over the 20 chromosomes runs
**0.3125-0.9558**, median **0.8015**, mean 0.7769. So the two agree on the
variance split to within a couple of percent -- 0.80 against 0.82 -- which
is the answer this comparison was for: the engines were never in
disagreement about heritability, only about which chromosome's kinship to
hold the rest of the genome out from. Quote GEMMA's `pve` with its range
(it is per chromosome), BulkLMM's `h2` as one number (it is global), and do
not compare the p-values: `--loco` weights each chromosome differently, so
neither set is the other's null.
**What would settle it, in order of cost:**
* **GEMMA's own variance components — available, one flag away.** Tested on a
synthetic BIMBAM trio (40 individuals, 60 markers) with GEMMA 0.98.6: `-lmm 9
-pve` is accepted and writes the block
## pve estimate in the null model = 1.62846e-06
## se(pve) in the null model = -nan
## vg estimate in the null model = 0
## ve estimate in the null model = 0
So the flag works with `-lmm 9`; what was missing was the plumbing, and that
is now in. `gemma-wrapper` 5eb53fa copies each run's `.log.txt` **beside** the
archive (they were only inside the `.tar.xz`, so reading `pve` meant unpacking
a multi-megabyte archive), and `/compute/gemma` passes `-pve` on the GWA step
and reports the components per chromosome:
> GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451,
> chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to 0.9549
> over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg 23.3659, ve
> 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933, ve 0.8661).
every one of the twenty (only the ends shown here), because under `--loco`
each belongs to a different kinship matrix and a caller told only the range
cannot ask which chromosome is which. The unrounded values also go to the run
log as `variance:` lines, so a `?id=` run's
`/compute/gemma/status` stream carries them while the run is in flight.
(BXD::10001, 34 animals, LOCO, `-lmm 9`.) That is the number the 16627/16769
comparison lacked: GEMMA's own `h2`, per chromosome, objective function
named. Where the two engines still disagree, the disagreement is now between
two printed variance components rather than between a number and an
inference.
Two caveats, both measured. GEMMA prints `-nan` for `se(pve)` when the
estimate sits on a boundary, and this build reports that as "not available"
rather than as a number; a boundary pve is a direction, not a proportion — the
same caveat as BulkLMM's `h2`, from the other engine. And the components are
on the phenotype's own scale, so a trait measured in thousandths gives
variances in thousandths: BXD::16769 (values 0.0012-0.0122, 36 animals) yields
`pve 1.0000 ± 0.0016 (vg 3.411e-5, ve 3.411e-10)` — pve is scale-free and
still reads 1.0, but the absolute numbers are meaningless to compare across
traits. Compare `pve` with `h2`, not `vg` with `σ²_e`.
* **BulkLMM's likelihood over an h2 grid.** The package has
`BulkLMM.profile_LL(y::Matrix, G, covar, K, h2_grid, markerID; reml)` (internal,
not exported), returning `(ll_list_null, ll_list_alt)` over the grid. It wants
the phenotype as a one-column matrix and a covariate matrix (an all-zero column
is what `scan` uses with no covariates). Printing that profile at the peak
marker says whether an h2 of 0.000 is the maximum or the flat edge of one — the
difference between "no heritability" and "the estimator cannot tell".
* **The kinship matrices, or a summary.** BulkLMM's `calcKinship(g)` is one line
and can be written out per panel; GEMMA's is already cached as a `-gk`
`*.k.json`. Correlating their off-diagonals (or reporting min/median/max) is
enough to say whether the two engines disagree about *relatedness* before
arguing about what the variance components mean.
Until then, the honest reading is the one the analysis reached: BulkLMM's `h2`
and the thresholds that depend on it are approximate where `h2` is on a
boundary, and GEMMA's p-values for 16627 come from a kinship-dominated model —
its peak should not be compared with a fixed-effects LOD.
## Cores: `scan()` is serial by design
This was measured, not assumed. `Threads.@threads` appears in `bulkscan.jl` and
`bulkscan_helpers.jl` — **never in `scan.jl`**. The single-trait `scan()` this
endpoint calls runs on one Julia thread; the parallel speed-up that *is* visible
comes from BLAS:
```
--threads 1 scan 13.7 s BLAS 96 threads scan 10.5 s
--threads 4 scan 12.3 s BLAS 4 threads scan 7.7 s (better!)
```
OpenBLAS defaults to every core on the machine (96 here) and oversubscribes on
matrices of this shape. `scripts/bulklmm_scan.jl` therefore:
* **splits the markers into independent blocks** and scans them on Julia
threads (`--blocks`, `BULKLMM_BLOCKS`, default 4; `1` is a plain serial scan). It is *exact*: each marker's
LOD is independent once the null variance components are fixed, and those
depend on `y` and `K` only. Verified identical output — LOD 9.589938 and the
same thresholds at 1 and 4 blocks — and the permutation thresholds are
combined by taking each permutation's maximum across blocks before the
quantiles, which is precisely what a genome-wide threshold is;
* **caches the parsed matrix and the kinship** per panel (`read 1.3 → 0.4 s`,
kinship `0.9 → 0`);
* **caps BLAS** at 4 threads when serial, 1 per block when parallel;
* draws two reference lines: the genome-wide alpha 0.05 and the stricter
alpha 0.01. The reply reports alpha 0.10 as well, but that is the loosest
bar in the set, so it is drawn only on request (`?all_thresholds=true`, or an
explicit `?thresholds=`), and GEMMA's Bonferroni line on `/plot/gemma` is
opt-in in the same spirit;
* reports what ran in the reply: `julia threads 4, blocks 4, BLAS threads 96 |
read 1.5 s, scan 10.5 s`.
What that buys at this size: 20.9 s (1 block) → 18.3 s (4 blocks). Modest,
because the LMM loop is ~8 s of the run, each block re-estimates its variance
components, and Julia's startup is the rest. **Where cores pay is
`bulkscan()`** — many traits in one process, threaded over trait blocks — which
is the shape an eQTL endpoint wants and the next thing to add here.
## Configuration and the Julia side
| key | meaning |
|---|---|
| `BULKLMM_CMD` | the `julia` binary (default: the store's `julia-1.8.5`) |
| `BULKLMM_ENV` | the project its packages are instantiated into |
| `BULKLMM_DEPOT` | the depot those packages live in (default `$HOME/.julia`) |
| `BULKLMM_SCRIPT` | `scripts/bulklmm_scan.jl` |
| `BULKLMM_THREADS` | Julia threads (`--threads`), default **4** |
| `BULKLMM_BLOCKS` | independent marker blocks scanned in parallel, default **4** (`1` = serial scan, same answer) |
| `BULKLMM_CACHE_DIR` | scratch: prepared genotypes, cached matrix and kinship |
| `BULKLMM_PLOTTER` | `scripts/plot-bulklmm.py` for `POST /plot/bulklmm` |
The depot is small and deliberately so — **79 packages, 139 MB, no GPU/CUDA
stack**: BulkLMM's closure is CSV, DataFrames, Distributions, Optim and their
dependencies. Every request runs with `JULIA_PKG_OFFLINE=true`, so a scan can
never fetch a package; if something is missing, the run fails saying which
package, and the fix is to prepare the depot — a deliberate operator action,
not a side effect of a request.
**Packaging, current state.** The deployment uses the store's own
`julia-1.8.5` with a depot under `~/.julia`, and the project (`BULKLMM_ENV`)
pinned by `Manifest.toml` (BulkLMM 1.2.2 + CSV + DataFrames). `guix-bioinformatics`
already carries a working `julia-bin` (1.10.9) and a `julia-build-system`, but
of BulkLMM's closure only `julia-distributions`, `julia-optim` and
`julia-nlsolversbase` are packaged there — a pure-Guix build would need ~35–45
new package recipes. The pragmatic route, if the depot should stop being
`~/.julia`, is `julia-bin` plus a depot built once from our `Manifest.toml`
(`scripts/prepare-julia-depot.sh`) and pointed at by `BULKLMM_DEPOT`; the
alternative, `PackageCompiler`, would bake the driver into a single executable
and remove Julia startup from the request path entirely.
## Traps worth remembering
* `--project=<path>`, **not** `--project <path>`: Julia 1.8 takes the
space-separated form as a program argument and the run produces nothing, with
exit status 0.
* A `NaN` LOD parses as a valid `f64` in Rust; the parser filters non-finite
values, or a marker the model could not fit would be reported as a LOD of 0.
* The gate gives a caller one BulkLMM scan at a time (`compute_busy`), which is
why two tests that scan concurrently must tolerate a 409 rather than fail.
## Deployed state (2026-11-11)
Both processes were restarted with `GEMMA_WRAPPER_CMD` pointing at
`gemma-wrapper-1.00-pre1-4.5eb53fa` (the package is pinned to `5eb53fa`,
revision 4, in `guix-bioinformatics`' `gn/packages/gemma.scm`), because the
profile's own wrapper is older than the pin. Verified against the running
production instance rather than a fixture: `POST /compute/gemma` for
BXD::10001 returns `pve 0.3125 to 0.9558 over 20 chromosomes` in its notes.
|