| Age | Commit message (Collapse) | Author |
|
DOL (200 individuals, 112 913 markers, one genotype version) failed on
R/qtl with "There are missing chromosome IDs" and on GEMMA with a bare
wrapper bail-out. Two unrelated causes, both in our own code.
**R/qtl: a genotype code collided with a chromosome name.** DOL's header
says `@unk:X` -- X is the code for a missing genotype -- and the wrapper
passed that to `read.csv(na.strings=...)` for the *whole* file, so every
marker on the X chromosome lost its chromosome ID and `read.cross.csv`
refused the map. The unknown code now applies to the genotype columns
only. DOL scans: 112 913 markers in 42 s, where before it never got past
reading the file. Same fix in scripts/qtl_fitqtl.R, which shares the read.
**GEMMA: its leave-one-chromosome-out run cannot fit the Y.** With LOCO the
DOL panel is scanned per chromosome, and the Y run dies inside BLAS --
`Enforce failed for N>0 in src/fastblas.cpp ... fast_cblas_dgemm` -- taking
the whole request with it (a 502 whose detail is the wrapper's banner).
The mitochondrial chromosome has the same shape of problem. `prepare` for
the GEMMA engine now writes the bimbam pair without the Y and M markers
(cached beside the individual subset, line-aligned so the geno and snps
stay in step), and the note says what was left out: "46 Y/M markers left
out (GEMMA's LOCO cannot fit them)". DOL maps: 106 515 rows in 49 s.
**And the four tests that had been failing since before this work**, all
stale expectations rather than product bugs:
* `benchmark_questions` pinned two traits to "page 2" of a 100-per-page
search when they rank ~190th and ~219th of 246 -- one re-index pushes
them over. It now walks up to three pages and asserts presence and LOD,
not the page; and the helper `search()` pages, because a benchmark row
can name a trait that ranks outside the first hundred of its own query.
* `no_db_mode_degrades_gracefully` asserted exactly one hit for "alcohol"
from the index alone. The index grew; the behaviour under test is that
the search still answers, within the limit, with the database-backed
fields null. That is what it asserts now.
* `search_proxy_scenarios` and `search_parity_replay` never pinned the
engine they test, so an ambient `SEARCH_ENGINE=native` made the proxy
scenarios answer from the local index: a 200 where a 422 was expected,
and "expected 2 upstream calls, got 0" for every replayed query. Both now
set `SEARCH_ENGINE=forward`.
* src/genotypes.rs: `DEGENERATE_CHROMOSOMES`, `drop_degenerate_chromosomes`,
`Prepared.markers_dropped`.
* src/routes/gemma.rs: also the 500 that `eb5320d` introduced -- counting
parents/F1s from `panel - keep` instead of subtracting the exclusion
*set*, whose length can exceed the panel (subtract overflow).
* scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: na.strings scoped to the
genotype columns.
* tests/{benchmark,no_db,search,search_parity}.rs: as above.
Verified through the API on DOL trait 10003: R/qtl 112 913 rows, GEMMA
106 515 rows. gemma/rqtl/fitqtl suites green (14, 10, 4).
|
|
44 of this deployment's `.geno` files carry `cM` and no `Mb` column, and
none of them could be scanned. GeneNetwork3's wrapper (copied verbatim
here) assumes the position columns are `Chr, Locus, cM, Mb` and then starts
the samples at a hardcoded column 5. On a cM-only file that is one column
too far, so the genotype block loses its first individual and the whole scan
dies in `do.call(rbind, ...)` with "number of columns of result is not a
multiple of vector length" -- a message that says nothing about the cause.
The wrapper now picks the map column from whichever of the requested scale
and the file's own columns exists (falling back, and saying so), and starts
the samples right after the position columns; the sex default and the
sample-count guard use the same count. `scripts/qtl_fitqtl.R` had the same
assumption and got the same fix.
This is the second deliberate deviation from GeneNetwork3's script, after
the sample-count guard, and both are commented as such at the code.
* scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: the position column and the
first sample column are computed, not assumed.
* tests/test_data/rqtl_sample1_cm.geno: the existing fixture with `Mb`
dropped -- same panel, cM only.
* tests/rqtl.rs: `a_cm_only_genotype_file_maps` scans it and asserts all six
markers come back (the failure mode was five-vs-six columns, not an
error message that pointed at the genotype file).
* src/rqtl.rs, src/routes/rqtl.rs: the docs said the script never forwards
`--scale`; it does, and now falls back, so the request field says what
`pos` is in.
Verified through the API: DGRP (cM-only) now maps (3 markers), and AXB (499
markers) and BXD.4 (3 811) are unchanged.
307 tests pass; the same four as before do not.
|
|
`fitqtl` takes a `covar` argument, so the effect it reports can be the one
*after* conditioning -- which is usually the number worth quoting for a peak
that overlaps a known confounder. `covariates` now works on /compute/fitqtl
exactly as on /compute/rqtl: the same `covariates::resolve` (case attribute
by name, inline values, explicit map, same coding and same note), so the two
endpoints report a covariate identically.
The R side does what R/qtl needs: the phenotype file carries one column per
covariate after the trait, `--covarstruct` says which are categorical, and
the script dummy-expands those (k-1 against the first sorted level, missing
values left missing). Two things it has to get right:
* `model.matrix()` drops rows with a missing value, which silently misaligns
the covariate frame against the cross -- so the dummies are built directly
(`ifelse(is.na(x), NA, x == level)`), not through `model.matrix`;
* a covariate can be perfectly spread across the panel and **constant among
the individuals a trait covers** -- BXD's `Status` is exactly that -- and
R/qtl then stops with `Lapack routine dgesv: system is exactly singular`.
Columns that do not vary among the phenotyped individuals are left out,
and the reply names them, the same rule the GEMMA covariate file applies.
Measured on BXD trait 10001 at its peak, `BXD.4`:
no covariate LOD 2.9310 additive +2.394 32.8% var
with Status LOD 3.3602 additive +2.463 36.6% var
with an inline numeric covariate LOD 2.9373 additive +2.361
The LOD is no longer the scan's once a covariate is fitted -- it is a
different model -- which the docs now say, next to the no-covariate parity
check that still holds.
* scripts/qtl_fitqtl.R: `--covarstruct`, dummy expansion, the degenerate
column rule, and the dropped names in the summary line.
* src/fitqtl.rs: writes the covariates into the phenotype file and the
struct file, and passes both.
* src/routes/fitqtl.rs: the `covariates` field, resolved after alignment so
the coding and the complete-case count match /compute/rqtl.
* tests/fitqtl.rs: a covariate run (the note, the fitted row, a changed LOD)
and an unknown attribute refused with the list of valid ones.
* README.md, docs/www/SKILL.md: covariates on this endpoint, and why its LOD
stops matching the scan's.
306 tests pass; the same four as before do not.
|
|
`/compute/rqtl` answers "where" -- scanone gives a LOD per marker -- but not
"how much": a scan statistic is not a fitted model. R/qtl's `fitqtl` fits
one, and reports the QTL's effect, its standard error, the model LOD and the
variance explained. That is what a table of effects needs, and it is a
separate endpoint because it is a separate question, asked about a handful
of markers (a scan's peaks) rather than a genome: `fitqtl` fits one model
per marker, so a genome-wide version would be thousands of fits. Cap: 100
markers per request.
Rows are positional:
marker, chr, pos, lod, pct_var, additive, se, t, p
`additive` is fitqtl's estimate for the QTL term -- R/qtl codes the QTL
genotype +-1, so it is half the difference between the two homozygote
classes, the same convention GeneNetwork's published `additive` uses.
**Two independent checks that the numbers are right.** On BXD trait 10001
(`BXD.4`), at the scan's peak:
* `lod` = 2.930985, and `/compute/rqtl`'s scanone gives 2.930985 at the same
marker -- identical to every digit the CSV carries, because they are the
same cross and the same phenotype. That parity is now an integration test;
* `additive` = 2.394472, against the value genenetwork.org publishes for that
trait (`/api/v_pre1/trait/BXD/10001` -> 2.394444), which is 28 ppm away --
and that published number came from qtlreaper, a different engine.
* scripts/qtl_fitqtl.R (new): reads the same `.geno` and two-column
phenotype file as `rqtl_wrapper.R` (which is a verbatim copy of GN3's
script and so not extended), builds the cross the same way, fits
`fitqtl` at each named marker with `get.ests=TRUE`, and writes one CSV
row per marker -- plus a one-line summary naming the strongest fit.
`calc.genoprob(step=0)`: marker-level probabilities are all a
marker-level fit needs.
* src/fitqtl.rs (new): runs it, parses the CSV into positional rows (`chr`
stays text, it can be `X`), keeps the command line for `notes`.
* src/routes/fitqtl.rs (new): the endpoint -- same alignment rules as
`/compute/rqtl` (by name onto the panel, parents/F1s excluded unless
asked), the same `normalize`/`winsor_*` pipeline (effects come out on
that scale), and the same R/qtl gate, so a fitqtl request and a scan
queue against one another rather than competing for cores.
* src/config.rs: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`).
* tests/fitqtl.rs: the fixture (columns, sign of the effect, notes), the
LOD parity against scanone, and the refusals (no markers, both genotype
inputs).
* README.md, docs/www/SKILL.md: the endpoint, what each column means, and
when to reach for it instead of the scan.
305 tests pass; the same four as before do not.
|