| Age | Commit message (Collapse) | Author |
|
POST /compute/qtl2 gains nperm, perm_seed and perm_strata, and the reply
gains meta.permutation {n, seed, significant, suggestive, values} -- the
genome-wide maximum of every permutation, so a histogram needs no second
call. `data` stays the positional rows array on purpose: /plot/rqtl posts
it straight back and the monitor counts its rows, so the null went beside
the rows rather than around them (a generic Envelope::with_meta/Meta::extra,
skip_serializing_if, so no other endpoint's response changes).
Thresholds come from R/qtl2's scan1perm over the same genotype
probabilities, kinship and covariates as the scan, with chr_lengths from
the map -- without those it derives chromosome lengths from marker
positions and the numbers stop being comparable with R/qtl's. The
permutation CSV is written in R/qtl's two-column shape so one parser and
one percentile routine serve both engines: PermutationResult::from_values
is now shared with /compute/rqtl instead of duplicated.
Semantics, each with the note that says it:
* scan1perm shuffles phenotypes across individuals -- right for an
outbred cross, only approximate for a selfing RIL panel, where qtl2perm
is the correct package (not installed here, and not packaged);
* perm_strata takes a case attribute name (sex, Epoch, ...) and permutes
within it, because shuffling across sexes inflates the null;
* fewer than 100 permutations gets a note that the 0.05 quantile is a
quantile of too few values; more than 10 000 is refused, since each
permutation is a whole genome scan;
* a seed is reported always -- without one, two identical requests would
give different thresholds.
Cost, measured on BXD trait 10001 (34 phenotyped, 20 chromosomes, LOCO
kinship, 4 cores): 20 permutations 0.6 s, 100 in 2.2 s, 500 in 14 s,
1000 in 24 s through the API; 0.022-0.03 s per permutation. Thresholds
converge as expected (0.05: 4.331 at 100 and 200 permutations, 4.374 at
1000 on the raw probe) and are reproducible: the same seed gives the same
numbers on 1 and 4 cores.
Four things that only surfaced by running it, all commented where they
bit: optparse keeps the dash (--perm-out is opt[["perm-out"]], so
opt$perm_out was always NULL and the file landed somewhere the caller did
not look); perm_strata must be a named character vector, not a factor;
the LOD in the CSV is the LAST numeric field (R/qtl's shape puts the
permutation index first, and a first-field parser reads that as the LOD);
and percentiles interpolate at (n-1)p/100, so 100 maxima give 0.6733, not
0.67.
bin/monitor-gn4 gains compute-qtl2-perm (nperm 10, judging
meta.permutation) so a deploy that loses the R path shows up as a red
check rather than a surprise.
|
|
A multiparent population cannot come from a .geno. That file carries two
codes per marker (B/D, A/H/B) -- a collapse, not a founder allele -- and
the collapse is not invertible, so DOL's 194 DO animals cannot be fitted
as a DO however much reference data appears. What can be is a cross that
ships its own qtl2 control file with founder genotypes, so the endpoint
now takes one by name:
{"rqtl2_cross": "do", "trait_name": "DO::10001 (WBC)", "samples": ...}
* resolve_control_file looks in genotype/rqtl2/{name}.json, then
{name}/{name}.json (do/do.json), then {name}.json under
GENOTYPE_FILES; prepare_control_cross reads the cross's pheno, maps
and founder_geno headers without converting anything. It reports
crosstype, founders and whether the cross is multiparent, and
refuses a multiparent crosstype with no founder_geno rather than
silently scanning it as biallelic.
* Samples are matched to the cross's own population by name and the
strays are counted in notes; rqtl2_cross cannot be combined with
group or genotype_file.
* The gate is given individuals x founders, because a multiparent scan
fits one coefficient per founder allele at every marker.
* Autosomes only now applies to shipped crosses as well: the DO's X is
left out (369 markers) instead of being fitted without a sex
covariate, and the script subsets with cross[, chr = autosomal].
Measured on the DO (WBC, 742 individuals, 8 founders, 6044 autosomal
markers): probabilities 174 s / 1.39 GB (cached), LOCO kinship 15 s
(cached), scan 21 s. scan1 runs on one core for a multiparent cross
because its forked path stalls on allele probabilities here -- 25+
minutes at 8 cores with no CPU consumed, against 21 s serial.
Two bugs found by running an actual published trait through it:
* do_pmap.csv is in Mb, not the bp qtl2 documents, so every DO peak was
reported at 0.0 Mb. The map's unit is now detected from its largest
position and printed.
* The phenotype file was read with the default whitespace separator, so
a label with a space ("DO::10001 (WBC)") split into three fields and
read.table stopped with "line 1 did not have 3 elements". Both the
pheno and covariate reads are tab-separated now.
The R error extraction also joins R's wrapped continuation line, so a
complaint reads as a sentence rather than "Error in scan(...) :".
|
|
R/qtl sits between the existing two scans: Haley-Knott on marker
genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with
mean-imputed genotypes. Neither can analyse a multiparent population,
and neither models relatedness from the actual genotype data. R/qtl2
does both -- HMM genotype probabilities, then an LMM with
leave-one-chromosome-out kinship -- and it is what names the third peak
on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24,
against GEMMA's -log10 p 7.15 there).
* src/qtl2.rs converts this deployment's own .geno into a qtl2 cross
under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with
the genotype-code map, the orientation and the paths written out
explicitly. The shipped genotype/rqtl2/ control files cannot be
used -- they point outside the data directory, declare no code map
(every genotype reads as missing) and no orientation (individuals
read as markers) -- and each of those produces a plausible flat
scan instead of an error, which is why the conversion is ours.
* Heterozygous calls are dropped on a riself panel, as R/qtl's
convert2riself does. Left in, qtl2 reads the cross as multiparent
and its scan1 takes a path the panel cannot satisfy ("Error in
scan1_pg ... : No individuals in common").
* Autosomes only: X needs a sex covariate to be fitted properly and
Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is
restricted the same way. The note says how many markers were left
out.
* scripts/qtl2_scan.R caches genotype probabilities and kinship by a
hash of the control file *and its directory*; two conversions of
the same panel can have identical control files and identical file
sizes, and reusing the first one's probabilities silently scans the
wrong cross. It also takes the individual names from whichever
shape the cross's pheno slot has -- a single-column pheno file
leaves it a plain matrix, and the resulting empty phenotype reads
as scan1's "No individuals in common".
* Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply
unchanged -- post the same samples as /compute/rqtl.
* tests/qtl2.rs: the fixture scans and its notes name the cross; BXD
trait 10001 peaks on chr8 under both scans; refusals carry
missing_genotype / invalid_kinship.
cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like
R/qtl; a DO cross costs minutes of genotype probabilities, paid once
per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
|
|
Phase 0 of the R/qtl2 work, so the engine can be exercised before any
endpoint exists. `r-qtl2` (0.42 in this channel) plus `r-digest`/`r-jsonlite`
go into the dev shell's propagated inputs; without them R has to be handed a
list of 594 store paths by hand to load qtl2 at all.
`scripts/qtl2_scan.R` is written for this API rather than copied from
GeneNetwork3's `scripts/rqtl2_wrapper.R`, which cannot be trusted as a
reference: its `"LMM"` and `"LOCO"` branches call `scan1` with identical
arguments, its covariates are a `covar <- NULL; # TODO` stub, and its only
tests mock the file-writing helpers -- nothing ever reads a cross.
The script reads a qtl2 control file, a two-column phenotype file and
optional covariates, and writes `marker,chr,pos,lod` (Mb, so the shape
matches /compute/rqtl and /plot/rqtl draws it unchanged). Genotype
probabilities and the kinship matrices are cached in --cache-dir keyed by
the control file's content, because a DO cross costs 536 s of genotype
probabilities on four cores against 11 s of kinship.
Four things the cross-file plumbing cost, all found by running it:
* `read_cross2` joins the control file's directory onto whatever paths it
finds, so an absolute path becomes `<dir>/<absolute>` and dies. The server
has to write the control file *beside* its data;
* our generated genotypes are markers-in-rows, which needs
`"geno_transposed": true` -- without it qtl2 reads the individuals as
markers (93 individuals appear as 93 extra markers, and every marker is
"not in both genotypes and maps");
* the genotype strings need a `"genotypes"` map (the GN letters
@mat/@het/@pat -> A/H/B -> 1/2/3); without it every value is "treated as
missing" and the scan comes back flat (peak LOD 0.14);
* with those fixed, a cross converted from `genotype/BXD.4.geno` scans in
~1 s and agrees with /compute/rqtl on the same panel and trait: qtl2's LMM
peaks on chr8 at ~111 Mb, LOD 3.50, where R/qtl's HK `scanone` peaks on
chr8 at ~69 Mb, LOD 2.93 -- same chromosome, different model, which is the
cross-check the plan asks for.
Not yet: the Rust side (`src/qtl2.rs`), the `/compute/qtl2` route, the
genotypes-listing `qtl2` flag and the converter that writes the control
file. Those are Phase 1-3.
|