| Age | Commit message (Collapse) | Author |
|
Parents and F1s are excluded by default (as in GeneNetwork2/3), with
include_parents_f1 to keep them all. That boolean was not enough for what was
asked -- "include these, drop that substrain" -- and the exclusion list itself
is data (parents_and_f1s.json, seven groups) that no request can reach. F1
names also differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH,
C57BL/6JxBXD037F1 for the intercross F1s), so pattern matching is the wrong
tool.
Every group-mapping endpoint -- /compute/gemma, /compute/gemma/ravanan,
/compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm -- now takes
"individuals": {"include": [...], "exclude": [...]}
applied after the parent/F1 rule, exclude winning over include. An unknown name
is a 422 invalid_individuals whose message lists the panel: a silently dropped
animal changes a kinship matrix without anyone noticing, so the failure has to
be loud. genotypes::select_panel is the one implementation, and it returns what
it did (dropped parents/F1s, re-included by name, excluded by name) so the
Genotype note can say it: "197 individuals, 38 parents/F1s dropped; 1 excluded
by name".
The note now also spells out the distinction that makes this option
treacherous: the kinship uses *every* individual in the genotype file, the
association uses only those with a trait value -- GEMMA reads no phenotypes for
the kinship. So "include the F1s" changes the relatedness structure without
putting the F1s in the fit, and a request can legitimately have 234 individuals
in the genotype file and 34 in the association. Verified on BXD.8: excluding
BXD1 gives 197 individuals / 33 values; include_parents_f1 with an F1
re-included and a substrain excluded gives 234 / 34.
A basket supplies values by name, so a selection applies to a basket-fed
mapping the same way: the individuals excluded are simply not in the panel any
more, and their basket values are ignored.
GenotypeError gains Invalid for this (422) so a caller's typo is not reported
as a missing genotype file; the read-only endpoints that can never see one map
it the same way for exhaustiveness.
|
|
A caller that knows only /version should be able to reach anything by
following links. Until now `next` was thin and uneven: the genotype
subtree offered none at all, the group record offered one, /version
offered nothing, and two links it did offer could not be followed.
* suggest:: now builds the graph: root_links (groups, species,
mapping-methods, openapi), genotype_version_links (frequencies,
marker names, a marker's location and tally, the group, the scanning
actions), unlisted_genotype_links, marker_links (location, tally,
marker list, panel tally, PheWAS at that marker), group_links and
trait_links. Envelope::with_links / with_links_except add them,
deduplicating and never suggesting the response the caller is
already reading.
* Wired through the genotype surface (listing, version detail, markers,
location), both frequencies routes, the compute endpoints, groups,
traits, both value routes, species, datasets, publications,
accessions, search, mapping methods, phewas, correlation and
/version.
* Fixed what was not followable, because `next` promises a GET that
returns 200 and a client is the one that finds out otherwise:
- /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are
gone; comput endpoints now link ?locus=<their strongest marker>
(suggest::peak_marker_of, computed before the rows move).
- /datasets and /publications are not routes; root_links offers the
listings that exist (/groups, /species, /mapping-methods) and
/openapi.json.
- /search?q= with an empty query is a 400, so it is not a link.
- the correlation endpoint's "/compute/correlation" was a POST in
`next`; it is in `actions` now.
- marker_links pointed at /markers/{marker}, which is not a route;
it points at /markers/{marker}/location.
tests/links.rs walks the graph two hops from a dozen seeds and fails on
any link that does not answer 200, plus a check that a tally leads to the
marker list and to a marker's location, and a location to that marker's
tally and its PheWAS.
|
|
Two questions the genotype surface could not answer: what a panel is
actually called as, and which markers it holds. The listing gives counts
and the compute endpoints give scans; a panel whose calls are not the
codes its header declares, or a marker that is 99% missing, stayed
invisible until a scan came back flat -- and then the search for the
cause started at the wrong end.
* GET /groups/{group}/genotypes/{version}/frequencies counts the calls
in the version's .geno and returns them as call -> frequency
({"B": 0.4889, "D": 0.492109, "H": 0.018991}). Frequencies are over
the calls that are present, so they sum to 1; missing codes (U, N,
NA, an empty cell) are counted separately and reported in notes,
because the missing rate is the diagnosis.
* A probability-valued file is binned 0.05 wide (twenty bins over
[0, 1], keys are the bin edges), since 0.4187 and 0.4191 are the same
statement about a marker; ?bin= sets another width. A file with a few
distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported
exactly -- binning three values hides what is being looked at.
* .../markers/{marker}/frequencies is one locus, and ?marker= on the
panel route is the same call. Counting goes through an interner, so
cost is one hash lookup per cell and one allocation per distinct
call; a .geno over 256 MB is refused (413 genotype_too_large) rather
than read, which is what HSNIH-Palmer_true (3.7 GB, 134927 markers x
6147 individuals) demanded.
* The version's /markers now lists marker NAMES only, one column, and
.../markers/{marker}/location returns {"name","chr","cM","Mb"} as
named fields: a page of names stays small, and a position is a
lookup rather than a table the client must hold. The same size guard
applies, so an oversized panel's listing is a 413 in 3 ms.
* Files no group lists -- a panel in preparation, a fixture, the DO's
do/ cross -- live at /groups/unlisted/genotypes/{name}/... with the
same four calls. `unlisted` is a namespace, not a group: only its
genotype subtree answers. One prefix for the whole genotype surface
beats a second spelling for the same tally, so the earlier
/genotypes/{name} spelling is gone.
* genotypes::resolve_named (moved out of the frequencies module, since
it is genotype metadata) resolves a version, a .geno name, or a path
relative to the genotype files directory; MAX_SCAN_BYTES is shared by
every whole-file read.
Tests limited to the feature: genotypes 3 (names-only columns, paging,
location fields, marker_not_found, unlisted parity, 413), frequencies 5,
frequencies unit 7.
|
|
The listing named versions but not files; a caller could pick one and
still not see what was in it. Three token-gated routes now describe a
version completely, short of the matrix:
GET /groups/{group}/genotypes versions, files, sizes, current
GET /groups/{group}/genotypes/{version} file name, header labels,
individuals, strains, chromosomes
GET /groups/{group}/genotypes/{version}/markers markers (name, chr, cM, Mb), paged
The detail reads the file's own `@`-labels (type, mat, pat, het, unk,
scale) and its panel in column order, and reports both the whole panel
and the strains a mapping uses after the parents/F1s rule -- so a caller
can check a request before sending it and interpret a result after.
Markers come back as positional rows with `meta.columns`, paged with
`?page`/`?limit` and filterable with `?chr`, following /compute/gemma's
shape rather than inventing one.
The genotype calls stay on the server: file_meta() reads the matrix to
skip it, and nothing returns a cell.
* src/genotypes.rs (Marker, FileMeta, file_meta; GenotypeFile::file):
header labels, individuals and marker positions; unit test.
* src/routes/genotypes.rs: the two new handlers, token-gated like the
listing, plus `file` in the listing and counts in the detail.
* tests/genotypes.rs: the listing carries file names but no paths.
* README.md, docs/genotypes.md, docs/www/ai-instruction.md: what the
three routes give, and that the matrix is not among them.
|
|
Until now /compute/{rqtl,gemma} took server paths, so the caller had to
know the deployment's layout, pick a version, and expand a trait's values
onto the panel in the file's order -- none of which the API told it. The
listing endpoint and the group-scoped request remove all three:
GET /groups/{group}/genotypes versions, newest first, no paths
POST /compute/rqtl {group, [genotype_version], samples}
POST /compute/gemma {group, [genotype_version], samples}
The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the
group's genotype/<Group>.json `genofile` list is newest-first and its
first entry is current; groups with no JSON (HET3-ITP) fall back to the
highest version number, and the listing's notes say which happened --
reads the panel from the .geno header the way GN3 does, and excludes
parents and F1s by default (GN2's `incparentsf1` off, GN3's
`get_sample_list(inc_par=False, inc_f1=False)`).
Excluding them has to subset the genotype matrix, not null-fill the
values: a null keeps an individual out of the fit but not out of GEMMA's
kinship or R/qtl's cross. So `prepare` writes a panel-subset file under
the engine's cache directory, keyed by panel, and reuses it.
Nothing in the listing or the mapping response carries a path: the
endpoint reports versions and sizes, and `notes` names the resolved
version, the panel size and how many values aligned. Explicit
`genotype_file`/`snps_file` remain as an override for files outside the
layout, and the two request shapes are mutually exclusive (422).
* src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1
exclusion, subset preparation, name alignment; 10 unit tests.
* src/routes/genotypes.rs: new endpoint, with a test that its list has no
path or file name in it.
* src/routes/{rqtl,gemma}.rs: group + genotype_version +
include_parents_f1; named `samples` aligned server-side; notes record
what resolved. gemma also accepts named samples where it used to insist
on positional trait_values.
* src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now
carry `group` and no placeholders; the dataset-scoped ones keep the
placeholder and say to add the group.
* src/routes/groups.rs: the group record points at /genotypes.
* src/config.rs: GENOTYPE_FILES (default the gn4-data directory).
* tests/genotypes.rs: new integration test (skips without the directory).
* README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the
versions, map by group" -- measured runs, the parent/F1 rule, the
subsetting requirement, and the restart list unchanged.
|