| Age | Commit message (Collapse) | Author |
|
The note reported the range and named the two chromosomes that bounded it,
which is not enough to answer the question the numbers exist for. Under
--loco each pve belongs to a *different* kinship matrix, and GEMMA's
per-chromosome .log.txt files live on the server -- so a caller told "0.31
to 0.95" could not ask which chromosome was which, or compare a single
chromosome against BulkLMM.
The note now lists all twenty, compactly, with the range and the two
extremes keeping their vg/ve:
GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451,
chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to
0.9549 over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg
23.3659, ve 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933,
ve 0.8661).
and streaming::append_lines puts the unrounded values in the run log as
variance: chr7 pve=0.954938 se_pve=0.357032 vg=76.0933 ve=0.86612
so a ?id= run's /compute/gemma/status stream carries them while the run is
still in flight, rather than only in a reply someone has to keep. Both
measured on BXD::10001 through the dev server before committing.
The values are printed with four significant figures (scientific when the
magnitude needs it), unchanged: rounding variance components to four
decimals was the earlier bug, and these are the same numbers a reader
compares with BulkLMM's h2.
Docs: README's /compute/gemma section, SKILL's gemma bullet ("quote
per-chromosome values, not an average: they are twenty different kinship
matrices") and docs/bulklmm.md's example.
|
|
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values
have no threshold we can compute: a GEMMA permutation repeats the whole
scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual
stand-in and it is the wrong null -- it shuffles phenotypes as if the
individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global
kinship matrix and permutes by rotating the data through the kinship's
eigen decomposition, so a threshold costs seconds.
POST /compute/bulklmm {group, genotype_version, samples|basket,
nperm, seed, reml, assumption}
Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8,
35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2
4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81,
0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's
published locus and LOD, and a significant peak against a threshold this
API could not compute before.
Two differences from GEMMA are in every reply, not implied: a global
kinship matrix rather than leave-one-chromosome-out, and h2_null saying
how much relatedness was found (here ~0, so the fit is effectively
fixed-effects -- which is why it agrees with R/qtl where GEMMA differs).
* src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a
strains x markers matrix (our BIMBAM pair is already the allele
dosages BulkLMM wants), a marker map, the phenotype and covariates
-- runs the driver through streaming::run_streaming_with_env, and
reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid
f64 in Rust and would otherwise be reported as a value.
* scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds),
with the seed recorded so a threshold set is reproducible.
* gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is
cheap, unlike R/qtl's.
* Julia comes from the deployment's own store binary with a prepared
depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request
runs JULIA_PKG_OFFLINE=true, so a scan can never download anything.
`--project=<path>`, not `--project <path>`: Julia 1.8 takes the
space-separated form as a program argument.
tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and
seed are in the notes, and a scan without permutations says there is no
threshold rather than leaving a bare LOD to be read as significant.
|
|
mark_failed_appends_failure_marker_with_the_error and
create_run_log_rejects_a_reused_id both worked in
/tmp/gn4-streaming-test-<pid>, and the second removes the directory when
it is done, so under cargo's parallel test threads it could delete the
first one's log mid-assertion -- an intermittent failure that has
nothing to do with either test's subject. Give each its own directory.
|
|
- rqtl: per-call unique output filename. The content hash alone was
deterministic across processes, so two concurrent identical
requests shared one output file; R truncates on open, so one
request could read the other's half-written results with no error.
The per-call unique (pid + nanos, already computed for the pheno
file) now names the output file too.
- streaming/gemma/rqtl: append a failure marker to the run log on
every error path, mirroring the completion marker. A status
poller can now tell a failed run from a still-running one.
- rqtl: --scale is now honored. The wrapper selects the geno file's
cM/Mb map column from it, and the route rejects values other
than mb/cm with 422 instead of silently ignoring them.
Both new behaviors are unit-tested; the live Rscript tests pass.
|
|
Same request/response shape as /compute/gemma: synchronous (no job
queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming.
Extracted that streaming infrastructure (run_streaming, create_run_log/
RunLogError, status/StatusError, RunStatus/RunStatusParams,
is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new
src/streaming.rs so both compute endpoints share it instead of
duplicating it; src/routes/gemma.rs updated to the shared names,
confirmed its own 8 tests still pass unchanged.
scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim
and run via Rscript -- same reasoning as reusing gemma-wrapper rather
than reimplementing GEMMA. Scope: a single scan (scanone) plus, if
nperm > 0, its permutation thresholds (67th/95th percentile, via a
Rust port of numpy.percentile's default linear interpolation) -- the
common case. Not implemented: covariates, interval mapping,
permutation strata, pair-scan (scantwo) -- scantwo in particular needs
cross-referencing the genotype file for nearest-marker lookups and a
second (MAP_) output file, a materially bigger port than this covers.
One fix relative to GeneNetwork3, found by reading the code (no test
covers the real caller's keys): generate_rqtl_cmd's content hash only
ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller
uses keys "geno"/"pheno" -- they never match, so the genotype/
phenotype file contents never actually enter the cache-key hash in
production, only the other options. Two different phenotypes mapped
with the same other options would get the same cached output filename
and silently reuse each other's result. Fixed here: both files'
contents are always hashed (std DefaultHasher -- a cache key, not a
security boundary, so no need for GeneNetwork3's md5).
guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as
propagated-inputs (same category as gemma-wrapper, and same reasoning:
invoked as a subprocess, not linked). All stock Guix -- confirmed the
existing -L (needed only for gemma-wrapper, from guix-bioinformatics)
is not needed for these; the dev shell has Rscript/R-qtl on PATH from
guix.scm's own propagated-inputs alone.
Test fixture: a small, hand-written, valid BXD riset genotype/
phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's
own tests/unit/db/.../genotype_sample1.geno -- that fixture has a
genotype code outside its own header's declared set and duplicate
marker names across chromosomes, harmless for the unrelated parsing
unit it's written for there, but both would break a real R/qtl run.
Verified end to end, not just unit tested: tests/rqtl.rs runs the real
Rscript/R-qtl against that fixture (a real scanone scan, real
permutation thresholds with suggestive <= significant, the run-id-
collision 409 pre-seeded with no Rscript needed, and the completion
marker landing in a real log after the run finishes). cargo test
passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|