| Age | Commit message (Collapse) | Author |
|
The listing named versions but not files; a caller could pick one and
still not see what was in it. Three token-gated routes now describe a
version completely, short of the matrix:
GET /groups/{group}/genotypes versions, files, sizes, current
GET /groups/{group}/genotypes/{version} file name, header labels,
individuals, strains, chromosomes
GET /groups/{group}/genotypes/{version}/markers markers (name, chr, cM, Mb), paged
The detail reads the file's own `@`-labels (type, mat, pat, het, unk,
scale) and its panel in column order, and reports both the whole panel
and the strains a mapping uses after the parents/F1s rule -- so a caller
can check a request before sending it and interpret a result after.
Markers come back as positional rows with `meta.columns`, paged with
`?page`/`?limit` and filterable with `?chr`, following /compute/gemma's
shape rather than inventing one.
The genotype calls stay on the server: file_meta() reads the matrix to
skip it, and nothing returns a cell.
* src/genotypes.rs (Marker, FileMeta, file_meta; GenotypeFile::file):
header labels, individuals and marker positions; unit test.
* src/routes/genotypes.rs: the two new handlers, token-gated like the
listing, plus `file` in the listing and counts in the detail.
* tests/genotypes.rs: the listing carries file names but no paths.
* README.md, docs/genotypes.md, docs/www/ai-instruction.md: what the
three routes give, and that the matrix is not among them.
|
|
Until now /compute/{rqtl,gemma} took server paths, so the caller had to
know the deployment's layout, pick a version, and expand a trait's values
onto the panel in the file's order -- none of which the API told it. The
listing endpoint and the group-scoped request remove all three:
GET /groups/{group}/genotypes versions, newest first, no paths
POST /compute/rqtl {group, [genotype_version], samples}
POST /compute/gemma {group, [genotype_version], samples}
The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the
group's genotype/<Group>.json `genofile` list is newest-first and its
first entry is current; groups with no JSON (HET3-ITP) fall back to the
highest version number, and the listing's notes say which happened --
reads the panel from the .geno header the way GN3 does, and excludes
parents and F1s by default (GN2's `incparentsf1` off, GN3's
`get_sample_list(inc_par=False, inc_f1=False)`).
Excluding them has to subset the genotype matrix, not null-fill the
values: a null keeps an individual out of the fit but not out of GEMMA's
kinship or R/qtl's cross. So `prepare` writes a panel-subset file under
the engine's cache directory, keyed by panel, and reuses it.
Nothing in the listing or the mapping response carries a path: the
endpoint reports versions and sizes, and `notes` names the resolved
version, the panel size and how many values aligned. Explicit
`genotype_file`/`snps_file` remain as an override for files outside the
layout, and the two request shapes are mutually exclusive (422).
* src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1
exclusion, subset preparation, name alignment; 10 unit tests.
* src/routes/genotypes.rs: new endpoint, with a test that its list has no
path or file name in it.
* src/routes/{rqtl,gemma}.rs: group + genotype_version +
include_parents_f1; named `samples` aligned server-side; notes record
what resolved. gemma also accepts named samples where it used to insist
on positional trait_values.
* src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now
carry `group` and no placeholders; the dataset-scoped ones keep the
placeholder and say to add the group.
* src/routes/groups.rs: the group record points at /genotypes.
* src/config.rs: GENOTYPE_FILES (default the gn4-data directory).
* tests/genotypes.rs: new integration test (skips without the directory).
* README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the
versions, map by group" -- measured runs, the parent/F1 rule, the
subsetting requirement, and the restart list unchanged.
|
|
Paths travel in the request and no panel is cached in-process; both
engines key their caches on file contents (rqtl hashes the genotype file
into the output name, gemma-wrapper hashes into its kinship cache), so a
changed file is recomputed rather than silently reused.
The restart list is the useful half: new binary, GENENETWORK4_TEST_TOKEN
(decided when the router is built), and the environment fixed at first
use -- PATH/R_LIBS_SITE for R, SQL_URI's pool, the Xapian handle -- while
edits to scripts/rqtl_wrapper.R take effect immediately, since it runs
per request.
* docs/genotypes.md: "No restart is needed to use a new genotype file".
|
|
Asked whether the BXD mappings ran GN2/GN3's default (no parents, no
F1s): they did not. BXD.8's panel is 198 RI strains + 37 F1s and no
parent columns at all, and the runs passed all 235 with `null` for the
F1s -- phenotypes right, but GEMMA's kinship and R/qtl's cross spanning
235 individuals instead of 198.
Measured on trait 10001, both engines, both panels: identical answers
(GEMMA p 5.9e-07 at rsm10000005700 chr 8, R/qtl LOD 2.9320 at
rs48756159 chr 8), so nothing published from those runs is wrong. The
lesson is that null-filling excludes an individual from the fit but not
from the model; excluding it means subsetting the genotype file.
* docs/genotypes.md: "Parents and F1s are excluded by default" -- the
GN2 (`incparentsf1`) and GN3 (`get_sample_list(inc_par=False,
inc_f1=False)`) mechanisms, what each panel actually holds, the
both-ways measurement, and the rule for the alignment work.
* docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: the
same, in a sentence and a paragraph respectively.
|
|
docs/genotypes.md explained the convention; the endpoint docs still left
a caller guessing which file to pass and how big the panel is. So:
* README.md (/compute/gemma): GEMMA wants the BIMBAM pair
`genotype/bimbam/<file>_geno.txt` + `<file>_snps.txt`, not the `.geno`
file; `trait_values` is one entry per individual in that file's column
order; BXD's current panel stated (235 x 21056), with the 17.1 s
measurement against the 19 s the 7 320-marker fixtures take.
* README.md (/compute/rqtl): the file is `genotype/<file>.geno`
(BXD.8: 235 x 21057; HET3-ITP.12: 6447 x 894), `MappingMethodId`
decides the engine, and the two measured runs with the UM-HET3 peak.
* README.md (/compute/gemma/ravanan): same input convention as gemma.
* docs/genotypes.md: how each measurement was made, step by step, so
they can be repeated; and that step 3 (expanding values onto the panel)
is the API's job in the next iteration.
* docs/www/ai-instruction.md (4a): a table of the BXD files per engine
with panel sizes, the UM-HET3 file, "ask the operator for any other
panel" with where GN2 records which one is current, and the measured
times. Values must be one per individual, in file order -- position, not
name -- and the API does not re-index yet, so a mapping is prepared for
the caller rather than guessed.
* docs/www/operator-checklist.md: MappingMethodId per group, and the
three measurements.
|
|
The compute endpoints take server-side paths, so callers need to know
the deployment's genotype-file layout and which version is current --
neither of which is discoverable from this repository, and both of which
are spread across GN2 and GN3 (GN2's per-group genotype/<Group>.json
whose `genofile` list is newest-first and is the actual authority -- the
GenoFile table still lists only the classic files; GEMMA fed from
genotype/bimbam/<file>_geno.txt, R/qtl from genotype/<file>.geno; GN3
reading the sample order out of the .geno header; InbredSet.
MappingMethodId deciding which engine a group offers).
Measured on this deployment, with the newest genotypes and values
re-indexed onto each panel's sample order:
BXD BXD.8.geno (WGS Mar2022) GEMMA 17.1s 21055 rows
BXD BXD.8.geno R/qtl 7.5s 21056 markers
HET3-ITP.12 (Apr2023, 6447 individuals) R/qtl 83.1s 893 markers,
lifespan peak LOD 7.95 at 1_24042124 (chr1, 24.04 Mb)
That last run is the shape of what Rob Williams wants: UM-HET3 is a
4-way cross (MappingMethodId=3, R/qtl), and its genotype files are
versioned per cohort with no JSON recording which is current, and no
per-cohort trait split -- 6438 of lifespan's 23879 animal values fall in
the .12 panel, the rest are silently dropped. Recorded as the known
messiness rather than hidden.
* docs/genotypes.md: new. Layout, how GN2/GN3 choose, the alignment
rule, per-group engines, the measurements, what is still messy.
* README.md: pointer.
* docs/www/operator-checklist.md: replace "only the test fixtures exist"
with the real panels, and name the missing alignment step.
|