| Age | Commit message (Collapse) | Author |
|
GET /groups/{group}/parents-and-f1s answers the question the mapping notes only
implied. It reports the F1 strains, the maternal and paternal strain (from the
deployment's parents_and_f1s.json, ported from GN2's ParInfo -- all 24 groups),
the @type of the group's genotype file, the individuals each role names where
the metadata names them (BXD-Longevity does, most do not), and -- the part a
caller needs -- which members of the current panel the default rule drops and
which the `F1` name heuristic catches.
Two things that came out of building it and comparison with GN2:
* the JSON's paternal/maternal labels were INVERTED against GN2's reading.
GN2 unpacks ParInfo as `f1, f12, maternal, paternal`, so BXD's maternal
strain is C57BL/6J; our file said paternal. No mapping was affected
(non_strains takes both regardless of role) but the metadata was wrong, and
anything using "maternal" later -- reciprocal F1 direction, the @mat/@pat
note -- would have been backwards. The file is rewritten from ParInfo itself
(ast-parsed, not retyped), 23 groups' roles filled in and the 24th
(BXD-Longevity, which GN2 does not have) left as it was.
* an RI set has two kinds of F1 name: the design's (B6D2F1) and the panel's
own intercross individuals (C57BL/6JxBXD037F1). The endpoint reports
excluded_from_panel and excluded_by_name separately and says which is doing
the work, because "0 declared, 37 by name" is otherwise confusing. Empty F1
names (C57BL-6JxC57BL-6NJF2 has two) are dropped rather than becoming
exclusion entries that match nothing.
genotypes::non_strains is rebuilt on the same typed parser (DesignRole /
ParentalDesign), so the rule and the endpoint cannot disagree. tests: a unit
test for the roles, empty-name handling and the unlisted-group source, and an
integration test asserting BXD's maternal/paternal orientation and the note.
|
|
Parents and F1s are excluded by default (as in GeneNetwork2/3), with
include_parents_f1 to keep them all. That boolean was not enough for what was
asked -- "include these, drop that substrain" -- and the exclusion list itself
is data (parents_and_f1s.json, seven groups) that no request can reach. F1
names also differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH,
C57BL/6JxBXD037F1 for the intercross F1s), so pattern matching is the wrong
tool.
Every group-mapping endpoint -- /compute/gemma, /compute/gemma/ravanan,
/compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm -- now takes
"individuals": {"include": [...], "exclude": [...]}
applied after the parent/F1 rule, exclude winning over include. An unknown name
is a 422 invalid_individuals whose message lists the panel: a silently dropped
animal changes a kinship matrix without anyone noticing, so the failure has to
be loud. genotypes::select_panel is the one implementation, and it returns what
it did (dropped parents/F1s, re-included by name, excluded by name) so the
Genotype note can say it: "197 individuals, 38 parents/F1s dropped; 1 excluded
by name".
The note now also spells out the distinction that makes this option
treacherous: the kinship uses *every* individual in the genotype file, the
association uses only those with a trait value -- GEMMA reads no phenotypes for
the kinship. So "include the F1s" changes the relatedness structure without
putting the F1s in the fit, and a request can legitimately have 234 individuals
in the genotype file and 34 in the association. Verified on BXD.8: excluding
BXD1 gives 197 individuals / 33 values; include_parents_f1 with an F1
re-included and a substrain excluded gives 234 / 34.
A basket supplies values by name, so a selection applies to a basket-fed
mapping the same way: the individuals excluded are simply not in the panel any
more, and their basket values are ignored.
GenotypeError gains Invalid for this (422) so a caller's typo is not reported
as a missing genotype file; the read-only endpoints that can never see one map
it the same way for exhaustiveness.
|
|
Two things asked for, and one bug each.
* POST /uploads/genotypes with an existing name now REPLACES it. A
genotype is the user's file to keep, not a record the deployment
curates; the digest in every reply says which version ran.
* genotype_upload on all four engines -- /compute/{rqtl,gemma,qtl2,
fitqtl} -- in place of genotype_file/group. The server writes the
form each engine reads: a GN .geno for R/qtl and R/qtl2, a BIMBAM
pair (new: genoupload::to_bimbam, coding from the base's
@mat/@het/@pat, NA for missing, snps as marker/bp/chr) for GEMMA. No
user path travels with a request.
Verified against the full 3811-marker BXD.4 matrix uploaded as an update:
rqtl 3811 rows, gemma 3809, qtl2 3730, fitqtl by marker.
The bug worth naming: values given by name were being aligned against a
file, and for GEMMA that file is BIMBAM, which has no individual names --
so the phenotype file came out empty and the wrapper failed with a Ruby
backtrace that said nothing. GEMMA's values are now ordered by the
upload's own individual list before the pair is written. The related
fixes: rqtl and fitqtl align named values onto the file's individuals
(they previously demanded its column order), and genotypes::align_to_file
is the one place that does it.
tests/genoupload.rs: mapping an upload on rqtl/qtl2/fitqtl with values by
name, and replacing a name (one row in the listing, a new digest). GEMMA's
LOCO kinship needs more markers per chromosome than a small fixture has,
so it is exercised against a full panel rather than in the fixture.
|
|
Two questions the genotype surface could not answer: what a panel is
actually called as, and which markers it holds. The listing gives counts
and the compute endpoints give scans; a panel whose calls are not the
codes its header declares, or a marker that is 99% missing, stayed
invisible until a scan came back flat -- and then the search for the
cause started at the wrong end.
* GET /groups/{group}/genotypes/{version}/frequencies counts the calls
in the version's .geno and returns them as call -> frequency
({"B": 0.4889, "D": 0.492109, "H": 0.018991}). Frequencies are over
the calls that are present, so they sum to 1; missing codes (U, N,
NA, an empty cell) are counted separately and reported in notes,
because the missing rate is the diagnosis.
* A probability-valued file is binned 0.05 wide (twenty bins over
[0, 1], keys are the bin edges), since 0.4187 and 0.4191 are the same
statement about a marker; ?bin= sets another width. A file with a few
distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported
exactly -- binning three values hides what is being looked at.
* .../markers/{marker}/frequencies is one locus, and ?marker= on the
panel route is the same call. Counting goes through an interner, so
cost is one hash lookup per cell and one allocation per distinct
call; a .geno over 256 MB is refused (413 genotype_too_large) rather
than read, which is what HSNIH-Palmer_true (3.7 GB, 134927 markers x
6147 individuals) demanded.
* The version's /markers now lists marker NAMES only, one column, and
.../markers/{marker}/location returns {"name","chr","cM","Mb"} as
named fields: a page of names stays small, and a position is a
lookup rather than a table the client must hold. The same size guard
applies, so an oversized panel's listing is a 413 in 3 ms.
* Files no group lists -- a panel in preparation, a fixture, the DO's
do/ cross -- live at /groups/unlisted/genotypes/{name}/... with the
same four calls. `unlisted` is a namespace, not a group: only its
genotype subtree answers. One prefix for the whole genotype surface
beats a second spelling for the same tally, so the earlier
/genotypes/{name} spelling is gone.
* genotypes::resolve_named (moved out of the frequencies module, since
it is genotype metadata) resolves a version, a .geno name, or a path
relative to the genotype files directory; MAX_SCAN_BYTES is shared by
every whole-file read.
Tests limited to the feature: genotypes 3 (names-only columns, paging,
location fields, marker_not_found, unlisted parity, 413), frequencies 5,
frequencies unit 7.
|
|
R/qtl sits between the existing two scans: Haley-Knott on marker
genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with
mean-imputed genotypes. Neither can analyse a multiparent population,
and neither models relatedness from the actual genotype data. R/qtl2
does both -- HMM genotype probabilities, then an LMM with
leave-one-chromosome-out kinship -- and it is what names the third peak
on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24,
against GEMMA's -log10 p 7.15 there).
* src/qtl2.rs converts this deployment's own .geno into a qtl2 cross
under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with
the genotype-code map, the orientation and the paths written out
explicitly. The shipped genotype/rqtl2/ control files cannot be
used -- they point outside the data directory, declare no code map
(every genotype reads as missing) and no orientation (individuals
read as markers) -- and each of those produces a plausible flat
scan instead of an error, which is why the conversion is ours.
* Heterozygous calls are dropped on a riself panel, as R/qtl's
convert2riself does. Left in, qtl2 reads the cross as multiparent
and its scan1 takes a path the panel cannot satisfy ("Error in
scan1_pg ... : No individuals in common").
* Autosomes only: X needs a sex covariate to be fitted properly and
Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is
restricted the same way. The note says how many markers were left
out.
* scripts/qtl2_scan.R caches genotype probabilities and kinship by a
hash of the control file *and its directory*; two conversions of
the same panel can have identical control files and identical file
sizes, and reusing the first one's probabilities silently scans the
wrong cross. It also takes the individual names from whichever
shape the cross's pheno slot has -- a single-column pheno file
leaves it a plain matrix, and the resulting empty phenotype reads
as scan1's "No individuals in common".
* Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply
unchanged -- post the same samples as /compute/rqtl.
* tests/qtl2.rs: the fixture scans and its notes name the cross; BXD
trait 10001 peaks on chr8 under both scans; refusals carry
missing_genotype / invalid_kinship.
cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like
R/qtl; a DO cross costs minutes of genotype probabilities, paid once
per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
|
|
DOL (200 individuals, 112 913 markers, one genotype version) failed on
R/qtl with "There are missing chromosome IDs" and on GEMMA with a bare
wrapper bail-out. Two unrelated causes, both in our own code.
**R/qtl: a genotype code collided with a chromosome name.** DOL's header
says `@unk:X` -- X is the code for a missing genotype -- and the wrapper
passed that to `read.csv(na.strings=...)` for the *whole* file, so every
marker on the X chromosome lost its chromosome ID and `read.cross.csv`
refused the map. The unknown code now applies to the genotype columns
only. DOL scans: 112 913 markers in 42 s, where before it never got past
reading the file. Same fix in scripts/qtl_fitqtl.R, which shares the read.
**GEMMA: its leave-one-chromosome-out run cannot fit the Y.** With LOCO the
DOL panel is scanned per chromosome, and the Y run dies inside BLAS --
`Enforce failed for N>0 in src/fastblas.cpp ... fast_cblas_dgemm` -- taking
the whole request with it (a 502 whose detail is the wrapper's banner).
The mitochondrial chromosome has the same shape of problem. `prepare` for
the GEMMA engine now writes the bimbam pair without the Y and M markers
(cached beside the individual subset, line-aligned so the geno and snps
stay in step), and the note says what was left out: "46 Y/M markers left
out (GEMMA's LOCO cannot fit them)". DOL maps: 106 515 rows in 49 s.
**And the four tests that had been failing since before this work**, all
stale expectations rather than product bugs:
* `benchmark_questions` pinned two traits to "page 2" of a 100-per-page
search when they rank ~190th and ~219th of 246 -- one re-index pushes
them over. It now walks up to three pages and asserts presence and LOD,
not the page; and the helper `search()` pages, because a benchmark row
can name a trait that ranks outside the first hundred of its own query.
* `no_db_mode_degrades_gracefully` asserted exactly one hit for "alcohol"
from the index alone. The index grew; the behaviour under test is that
the search still answers, within the limit, with the database-backed
fields null. That is what it asserts now.
* `search_proxy_scenarios` and `search_parity_replay` never pinned the
engine they test, so an ambient `SEARCH_ENGINE=native` made the proxy
scenarios answer from the local index: a 200 where a 422 was expected,
and "expected 2 upstream calls, got 0" for every replayed query. Both now
set `SEARCH_ENGINE=forward`.
* src/genotypes.rs: `DEGENERATE_CHROMOSOMES`, `drop_degenerate_chromosomes`,
`Prepared.markers_dropped`.
* src/routes/gemma.rs: also the 500 that `eb5320d` introduced -- counting
parents/F1s from `panel - keep` instead of subtracting the exclusion
*set*, whose length can exceed the panel (subtract overflow).
* scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: na.strings scoped to the
genotype columns.
* tests/{benchmark,no_db,search,search_parity}.rs: as above.
Verified through the API on DOL trait 10003: R/qtl 112 913 rows, GEMMA
106 515 rows. gemma/rqtl/fitqtl suites green (14, 10, 4).
|
|
Asking whether the singular matrices could be fixed turned out to have two
answers, one of them a real bug on our side.
The bug: a panel can name individuals whose genotype row is empty, and an
empty row makes the kinship matrix singular. GEMMA then reports
"eigenvalues close to zero" and dies in LAPACK with
`LUDecomp ... input domain error` -- a message about linear algebra for a
problem that is a missing genotype. Measured on this deployment's own files:
BXD-AE.4 136 of 344 individuals have no genotypes
MDP 93 of 178
BXD-Heart-Metals.4 200 of 481
HET3-ITP.3 2867 of 2867 (the whole file is empty)
Dropping them makes those panels map: BXD-AE.4's GWA step goes from a
failure to 3.2 s, and MDP's kinship step from a failure to success. The
individuals contribute nothing to a scan either way, so nothing is lost;
`notes` counts them, and the "Genotype:" line no longer blames parents/F1s
for them (`prepared.excluded` is "panel minus kept", which now includes
both, so the two counts are separated).
Not the cause, but worth naming: `AXB` fails earlier still, on its
annotation -- `ERROR: Enforce failed for D1Mit296 in src/gemma_io.cpp ...
ReadFile_anno` -- which is a broken `.snps` file, not a matrix. That wants a
pre-flight check of the annotation (report the offending marker) rather
than a GEMMA failure, and is not in this commit.
Considered and rejected: a ridge on K (`K + epsilon*I`) would work
mathematically but GEMMA has no option for it and the wrapper caches K by
content hash, so it would mean patching the wrapper's cached matrix; `-gk 2`
and dropping markers do not add rank; and the non-LOCO path is the one that
does not run at all here.
* src/genotypes.rs: `individuals_with_genotypes`, one pass over the bimbam
file marking the individuals with any genotype at all.
* src/routes/gemma.rs: filter `keep` by it before alignment (so the trait
values drop with them), and say so in `notes`.
* tests/gemma.rs: `individuals_with_no_genotypes_are_left_out` -- BXD-AE.4
maps, the note names the drop and the reason, and the genotype line no
longer claims 344 individuals.
* README.md: the behaviour and why it exists, under /compute/gemma.
307 tests pass; the same four as before do not.
|
|
The listing named versions but not files; a caller could pick one and
still not see what was in it. Three token-gated routes now describe a
version completely, short of the matrix:
GET /groups/{group}/genotypes versions, files, sizes, current
GET /groups/{group}/genotypes/{version} file name, header labels,
individuals, strains, chromosomes
GET /groups/{group}/genotypes/{version}/markers markers (name, chr, cM, Mb), paged
The detail reads the file's own `@`-labels (type, mat, pat, het, unk,
scale) and its panel in column order, and reports both the whole panel
and the strains a mapping uses after the parents/F1s rule -- so a caller
can check a request before sending it and interpret a result after.
Markers come back as positional rows with `meta.columns`, paged with
`?page`/`?limit` and filterable with `?chr`, following /compute/gemma's
shape rather than inventing one.
The genotype calls stay on the server: file_meta() reads the matrix to
skip it, and nothing returns a cell.
* src/genotypes.rs (Marker, FileMeta, file_meta; GenotypeFile::file):
header labels, individuals and marker positions; unit test.
* src/routes/genotypes.rs: the two new handlers, token-gated like the
listing, plus `file` in the listing and counts in the detail.
* tests/genotypes.rs: the listing carries file names but no paths.
* README.md, docs/genotypes.md, docs/www/ai-instruction.md: what the
three routes give, and that the matrix is not among them.
|
|
Until now /compute/{rqtl,gemma} took server paths, so the caller had to
know the deployment's layout, pick a version, and expand a trait's values
onto the panel in the file's order -- none of which the API told it. The
listing endpoint and the group-scoped request remove all three:
GET /groups/{group}/genotypes versions, newest first, no paths
POST /compute/rqtl {group, [genotype_version], samples}
POST /compute/gemma {group, [genotype_version], samples}
The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the
group's genotype/<Group>.json `genofile` list is newest-first and its
first entry is current; groups with no JSON (HET3-ITP) fall back to the
highest version number, and the listing's notes say which happened --
reads the panel from the .geno header the way GN3 does, and excludes
parents and F1s by default (GN2's `incparentsf1` off, GN3's
`get_sample_list(inc_par=False, inc_f1=False)`).
Excluding them has to subset the genotype matrix, not null-fill the
values: a null keeps an individual out of the fit but not out of GEMMA's
kinship or R/qtl's cross. So `prepare` writes a panel-subset file under
the engine's cache directory, keyed by panel, and reuses it.
Nothing in the listing or the mapping response carries a path: the
endpoint reports versions and sizes, and `notes` names the resolved
version, the panel size and how many values aligned. Explicit
`genotype_file`/`snps_file` remain as an override for files outside the
layout, and the two request shapes are mutually exclusive (422).
* src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1
exclusion, subset preparation, name alignment; 10 unit tests.
* src/routes/genotypes.rs: new endpoint, with a test that its list has no
path or file name in it.
* src/routes/{rqtl,gemma}.rs: group + genotype_version +
include_parents_f1; named `samples` aligned server-side; notes record
what resolved. gemma also accepts named samples where it used to insist
on positional trait_values.
* src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now
carry `group` and no placeholders; the dataset-scoped ones keep the
placeholder and say to add the group.
* src/routes/groups.rs: the group record points at /genotypes.
* src/config.rs: GENOTYPE_FILES (default the gn4-data directory).
* tests/genotypes.rs: new integration test (skips without the directory).
* README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the
versions, map by group" -- measured runs, the parent/F1 rule, the
subsetting requirement, and the restart list unchanged.
|