about summary refs log tree commit diff
path: root/test/python/test_groups.py
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-11 12:37:26 +0000
committerPjotr <wrk@bosa.lan>2026-11-11 12:37:26 +0000
commit84aced61986f229931b5e7738b4c64df45251a85 (patch)
tree8561fd6c863be1ccf828716817e1a34cb8db0f67 /test/python/test_groups.py
parent3022797b55bae96322439770ea48e74eb2e0eb8d (diff)
downloadgenenetwork4-84aced61986f229931b5e7738b4c64df45251a85.tar.gz
bulklmm: exact marker-block parallelism, and what the cores are actually doing
Reported: a BulkLMM run does not look multicore. It is not a
misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint
calls -- has no `Threads.@threads` at all; the package threads
`bulkscan()` (many traits) and bulkscan_helpers, not scan. The
parallelism that was visible before was BLAS (OpenBLAS defaulting to all
96 cores here), which is why `user` time ran 4x `real` while Julia
threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s.

  * The driver splits the markers into independent blocks and scans them
    on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact:
    the LOD for each marker is independent given the null variance
    components, which depend on y and K only. Verified identical -- peak
    LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks --
    and permutation thresholds combine by taking each permutation's
    maximum across blocks before the quantiles, which is what a
    genome-wide threshold is.
  * The parsed matrix and the kinship are cached per panel (read 1.3 s ->
    0.4 s, kinship 0.9 s -> 0), because both depend only on the cross.
  * The summary in `notes` now says what ran: julia threads, blocks, BLAS
    threads, and the read/scan split.
  * The hand parser is back: CSV.jl into a DataFrame is an order of
    magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s).
  * BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for
    one scan).

Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks
18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of
the run and each block re-estimates its variance components. Where the
cores pay is bulkscan() -- many traits in one process, threaded over
trait blocks -- which is the shape the next endpoint wants; documented in
the README rather than left as folklore.
Diffstat (limited to 'test/python/test_groups.py')
0 files changed, 0 insertions, 0 deletions