about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-11 12:52:36 +0000
committerPjotr <wrk@bosa.lan>2026-11-11 12:52:36 +0000
commitc9db07f6d372b0b8812e5f6698257a9df94dc1ef (patch)
tree98b8e918ff8d7e2d6e97773bc6326fb4f7513721 /docs
parente616b84fc86b176a20170a61ad205979af8bffc1 (diff)
downloadgenenetwork4-c9db07f6d372b0b8812e5f6698257a9df94dc1ef.tar.gz
docs: what BulkLMM models, what its cores do, and how to package it
docs/bulklmm.md records the three things that are easy to get wrong when a
BulkLMM result is quoted:

  * where it stands against the other engines -- Haley-Knott with no
    kinship (R/qtl), LOCO LMM whose permutations are refused (GEMMA),
    LOCO LMM on HMM probabilities (R/qtl2), and this: a global-kinship
    LMM whose permutation thresholds are rotation-based and therefore
    cheap -- the only engine here that can say "genome-wide significant
    at alpha 0.05" against a null that respects relatedness;
  * what it is not: global kinship rather than LOCO, so its p-values are
    not GEMMA's; h2_null says how much relatedness it found (near zero
    here, which is why it reproduces R/qtl's 9.59 where GEMMA differs);
    one degree of freedom, so multiparent scans stay with /compute/qtl2;
    and no standard errors in 1.2.2's scan(), so the rows are
    marker, chr, pos, lod, beta, h2;
  * the cores, measured: Threads.@threads is in bulkscan.jl and
    bulkscan_helpers.jl, never in scan.jl, so a single-trait scan is
    serial by design and the visible parallelism was BLAS (96 threads
    10.5 s against 4 threads 7.7 s). What the driver does about it --
    exact marker blocks, cached matrix and kinship, capped BLAS, and the
    breakdown printed into the reply -- with the numbers (20.9 s at one
    block against 18.3 s at four, same LOD 9.589938).

Also the packaging picture, since it is not obvious from the outside:
guix-bioinformatics has a working julia-bin and a julia-build-system, but
only three of BulkLMM's closure are packaged, so a pure-Guix build is
35-45 recipes; the pragmatic route is julia-bin plus a depot built once
from the pinned Manifest. scripts/prepare-julia-depot.sh is that step,
written down, and the depot stays small and GPU-free (79 packages,
139 MB) with JULIA_PKG_OFFLINE=true at request time.

The plot test learns that both thresholds are drawn, not one.
Diffstat (limited to 'docs')
-rw-r--r--docs/bulklmm.md120
1 files changed, 120 insertions, 0 deletions
diff --git a/docs/bulklmm.md b/docs/bulklmm.md
new file mode 100644
index 0000000..cc6bc73
--- /dev/null
+++ b/docs/bulklmm.md
@@ -0,0 +1,120 @@
+# BulkLMM: an LMM scan whose thresholds we can actually compute
+
+`POST /compute/bulklmm` runs [BulkLMM.jl](https://github.com/senresearch/BulkLMM.jl)
+(Apache-2.0) through `scripts/bulklmm_scan.jl`. This note records why the engine
+is here, what it does and does not model, and what its cores are doing — three
+things that are easy to get wrong when a result is quoted.
+
+## Why it sits beside GEMMA and R/qtl
+
+| | model | kinship | permutations |
+|---|---|---|---|
+| `/compute/rqtl` | Haley-Knott (single marker) | none | yes, but shuffles phenotypes — the wrong null for related individuals |
+| `/compute/gemma` | LMM, LOCO | leave-one-chromosome-out | **refused**: each one re-runs the whole LOCO scan |
+| `/compute/qtl2` | LMM, LOCO, HMM probabilities | LOCO | no |
+| `/compute/bulklmm` | LMM, **global** kinship | one K for the whole genome | **yes, cheaply**: rotates by K's eigen decomposition |
+
+So BulkLMM is the only engine here that can say *"this peak is genome-wide
+significant at α = 0.05"* against a null that respects relatedness. That is what
+`/compute/gemma`'s `permutations_not_supported` refusal now points at.
+
+Worked example, BXD trait 16627 on `BXD.8` (35 phenotyped animals, 21 056
+markers):
+
+```
+peak rsm10000000888  chr2 4.92 Mb  LOD 9.59  beta 0.0027  h2_null ≈ 0
+thresholds (200 rotations, seed 42): alpha 0.10 → 3.51, 0.05 → 3.81, 0.01 → 4.47
+```
+
+`rsm10000000888` is the trait's published locus and 9.59 its published LOD, so
+the engine agrees with the record — and the peak clears the α = 0.01 threshold.
+
+## What it is not
+
+* **A global kinship matrix, not LOCO.** One K for every marker; GEMMA excludes
+  each chromosome in turn. Different p-values on the same data, and the reply
+  says so. Do not present a BulkLMM p-value as GEMMA's.
+* **`h2_null` is the honest tell.** It is the heritability estimated under the
+  null (intercept) model, from `y` and `K` alone. Near zero — as here — the fit
+  is effectively a fixed-effects model, which is exactly why it reproduces
+  R/qtl's answer on this trait where GEMMA's LOCO scan differs.
+* **One degree of freedom.** BulkLMM 1.2.2 tests one coefficient per marker; a
+  multiparent founder-allele model is `/compute/qtl2`'s job (`crosstype: do`).
+* **No standard errors yet.** `scan()` in 1.2.2 returns `(b, sigma2_e, h2_null,
+  lod, L_perms)`; `SE` is not among them, so the reply's `se` column does not
+  exist and the rows are `marker, chr, pos, lod, beta, h2`.
+
+## Cores: `scan()` is serial by design
+
+This was measured, not assumed. `Threads.@threads` appears in `bulkscan.jl` and
+`bulkscan_helpers.jl` — **never in `scan.jl`**. The single-trait `scan()` this
+endpoint calls runs on one Julia thread; the parallel speed-up that *is* visible
+comes from BLAS:
+
+```
+--threads 1  scan 13.7 s        BLAS 96 threads  scan 10.5 s
+--threads 4  scan 12.3 s        BLAS  4 threads  scan  7.7 s   (better!)
+```
+
+OpenBLAS defaults to every core on the machine (96 here) and oversubscribes on
+matrices of this shape. `scripts/bulklmm_scan.jl` therefore:
+
+* **splits the markers into independent blocks** and scans them on Julia
+  threads (`--blocks`, default `BULKLMM_THREADS`). It is *exact*: each marker's
+  LOD is independent once the null variance components are fixed, and those
+  depend on `y` and `K` only. Verified identical output — LOD 9.589938 and the
+  same thresholds at 1 and 4 blocks — and the permutation thresholds are
+  combined by taking each permutation's maximum across blocks before the
+  quantiles, which is precisely what a genome-wide threshold is;
+* **caches the parsed matrix and the kinship** per panel (`read 1.3 → 0.4 s`,
+  kinship `0.9 → 0`);
+* **caps BLAS** at 4 threads when serial, 1 per block when parallel;
+* reports what ran in the reply: `julia threads 4, blocks 4, BLAS threads 96 |
+  read 1.5 s, scan 10.5 s`.
+
+What that buys at this size: 20.9 s (1 block) → 18.3 s (4 blocks). Modest,
+because the LMM loop is ~8 s of the run, each block re-estimates its variance
+components, and Julia's startup is the rest. **Where cores pay is
+`bulkscan()`** — many traits in one process, threaded over trait blocks — which
+is the shape an eQTL endpoint wants and the next thing to add here.
+
+## Configuration and the Julia side
+
+| key | meaning |
+|---|---|
+| `BULKLMM_CMD` | the `julia` binary (default: the store's `julia-1.8.5`) |
+| `BULKLMM_ENV` | the project its packages are instantiated into |
+| `BULKLMM_DEPOT` | the depot those packages live in (default `$HOME/.julia`) |
+| `BULKLMM_SCRIPT` | `scripts/bulklmm_scan.jl` |
+| `BULKLMM_THREADS` | July threads / marker blocks (default 4) |
+| `BULKLMM_CACHE_DIR` | scratch: prepared genotypes, cached matrix and kinship |
+| `BULKLMM_PLOTTER` | `scripts/plot-bulklmm.py` for `POST /plot/bulklmm` |
+
+The depot is small and deliberately so — **79 packages, 139 MB, no GPU/CUDA
+stack**: BulkLMM's closure is CSV, DataFrames, Distributions, Optim and their
+dependencies. Every request runs with `JULIA_PKG_OFFLINE=true`, so a scan can
+never fetch a package; if something is missing, the run fails saying which
+package, and the fix is to prepare the depot — a deliberate operator action,
+not a side effect of a request.
+
+**Packaging, current state.** The deployment uses the store's own
+`julia-1.8.5` with a depot under `~/.julia`, and the project (`BULKLMM_ENV`)
+pinned by `Manifest.toml` (BulkLMM 1.2.2 + CSV + DataFrames). `guix-bioinformatics`
+already carries a working `julia-bin` (1.10.9) and a `julia-build-system`, but
+of BulkLMM's closure only `julia-distributions`, `julia-optim` and
+`julia-nlsolversbase` are packaged there — a pure-Guix build would need ~35–45
+new package recipes. The pragmatic route, if the depot should stop being
+`~/.julia`, is `julia-bin` plus a depot built once from our `Manifest.toml`
+(`scripts/prepare-julia-depot.sh`) and pointed at by `BULKLMM_DEPOT`; the
+alternative, `PackageCompiler`, would bake the driver into a single executable
+and remove Julia startup from the request path entirely.
+
+## Traps worth remembering
+
+* `--project=<path>`, **not** `--project <path>`: Julia 1.8 takes the
+  space-separated form as a program argument and the run produces nothing, with
+  exit status 0.
+* A `NaN` LOD parses as a valid `f64` in Rust; the parser filters non-finite
+  values, or a marker the model could not fit would be reported as a LOD of 0.
+* The gate gives a caller one BulkLMM scan at a time (`compute_busy`), which is
+  why two tests that scan concurrently must tolerate a 409 rather than fail.