about summary refs log tree commit diff
path: root/docs/genotypes.md
AgeCommit message (Collapse)Author
11 daysgenotypes: expose the metadata, never the genotype callsPjotr
The listing named versions but not files; a caller could pick one and still not see what was in it. Three token-gated routes now describe a version completely, short of the matrix: GET /groups/{group}/genotypes versions, files, sizes, current GET /groups/{group}/genotypes/{version} file name, header labels, individuals, strains, chromosomes GET /groups/{group}/genotypes/{version}/markers markers (name, chr, cM, Mb), paged The detail reads the file's own `@`-labels (type, mat, pat, het, unk, scale) and its panel in column order, and reports both the whole panel and the strains a mapping uses after the parents/F1s rule -- so a caller can check a request before sending it and interpret a result after. Markers come back as positional rows with `meta.columns`, paged with `?page`/`?limit` and filterable with `?chr`, following /compute/gemma's shape rather than inventing one. The genotype calls stay on the server: file_meta() reads the matrix to skip it, and nothing returns a cell. * src/genotypes.rs (Marker, FileMeta, file_meta; GenotypeFile::file): header labels, individuals and marker positions; unit test. * src/routes/genotypes.rs: the two new handlers, token-gated like the listing, plus `file` in the listing and counts in the detail. * tests/genotypes.rs: the listing carries file names but no paths. * README.md, docs/genotypes.md, docs/www/ai-instruction.md: what the three routes give, and that the matrix is not among them.
11 daysgenotypes: list a group's genotype files, and map by group + versionPjotr
Until now /compute/{rqtl,gemma} took server paths, so the caller had to know the deployment's layout, pick a version, and expand a trait's values onto the panel in the file's order -- none of which the API told it. The listing endpoint and the group-scoped request remove all three: GET /groups/{group}/genotypes versions, newest first, no paths POST /compute/rqtl {group, [genotype_version], samples} POST /compute/gemma {group, [genotype_version], samples} The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the group's genotype/<Group>.json `genofile` list is newest-first and its first entry is current; groups with no JSON (HET3-ITP) fall back to the highest version number, and the listing's notes say which happened -- reads the panel from the .geno header the way GN3 does, and excludes parents and F1s by default (GN2's `incparentsf1` off, GN3's `get_sample_list(inc_par=False, inc_f1=False)`). Excluding them has to subset the genotype matrix, not null-fill the values: a null keeps an individual out of the fit but not out of GEMMA's kinship or R/qtl's cross. So `prepare` writes a panel-subset file under the engine's cache directory, keyed by panel, and reuses it. Nothing in the listing or the mapping response carries a path: the endpoint reports versions and sizes, and `notes` names the resolved version, the panel size and how many values aligned. Explicit `genotype_file`/`snps_file` remain as an override for files outside the layout, and the two request shapes are mutually exclusive (422). * src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1 exclusion, subset preparation, name alignment; 10 unit tests. * src/routes/genotypes.rs: new endpoint, with a test that its list has no path or file name in it. * src/routes/{rqtl,gemma}.rs: group + genotype_version + include_parents_f1; named `samples` aligned server-side; notes record what resolved. gemma also accepts named samples where it used to insist on positional trait_values. * src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now carry `group` and no placeholders; the dataset-scoped ones keep the placeholder and say to add the group. * src/routes/groups.rs: the group record points at /genotypes. * src/config.rs: GENOTYPE_FILES (default the gn4-data directory). * tests/genotypes.rs: new integration test (skips without the directory). * README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the versions, map by group" -- measured runs, the parent/F1 rule, the subsetting requirement, and the restart list unchanged.
11 daysdocs: a new genotype file needs no restart; only code and env doPjotr
Paths travel in the request and no panel is cached in-process; both engines key their caches on file contents (rqtl hashes the genotype file into the output name, gemma-wrapper hashes into its kinship cache), so a changed file is recomputed rather than silently reused. The restart list is the useful half: new binary, GENENETWORK4_TEST_TOKEN (decided when the router is built), and the environment fixed at first use -- PATH/R_LIBS_SITE for R, SQL_URI's pool, the Xapian handle -- while edits to scripts/rqtl_wrapper.R take effect immediately, since it runs per request. * docs/genotypes.md: "No restart is needed to use a new genotype file".
11 daysdocs: parents and F1s are excluded by default, measured both waysPjotr
Asked whether the BXD mappings ran GN2/GN3's default (no parents, no F1s): they did not. BXD.8's panel is 198 RI strains + 37 F1s and no parent columns at all, and the runs passed all 235 with `null` for the F1s -- phenotypes right, but GEMMA's kinship and R/qtl's cross spanning 235 individuals instead of 198. Measured on trait 10001, both engines, both panels: identical answers (GEMMA p 5.9e-07 at rsm10000005700 chr 8, R/qtl LOD 2.9320 at rs48756159 chr 8), so nothing published from those runs is wrong. The lesson is that null-filling excludes an individual from the fit but not from the model; excluding it means subsetting the genotype file. * docs/genotypes.md: "Parents and F1s are excluded by default" -- the GN2 (`incparentsf1`) and GN3 (`get_sample_list(inc_par=False, inc_f1=False)`) mechanisms, what each panel actually holds, the both-ways measurement, and the rule for the alignment work. * docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: the same, in a sentence and a paragraph respectively.
11 daysdocs: name the real genotype files, panels and timings everywherePjotr
docs/genotypes.md explained the convention; the endpoint docs still left a caller guessing which file to pass and how big the panel is. So: * README.md (/compute/gemma): GEMMA wants the BIMBAM pair `genotype/bimbam/<file>_geno.txt` + `<file>_snps.txt`, not the `.geno` file; `trait_values` is one entry per individual in that file's column order; BXD's current panel stated (235 x 21056), with the 17.1 s measurement against the 19 s the 7 320-marker fixtures take. * README.md (/compute/rqtl): the file is `genotype/<file>.geno` (BXD.8: 235 x 21057; HET3-ITP.12: 6447 x 894), `MappingMethodId` decides the engine, and the two measured runs with the UM-HET3 peak. * README.md (/compute/gemma/ravanan): same input convention as gemma. * docs/genotypes.md: how each measurement was made, step by step, so they can be repeated; and that step 3 (expanding values onto the panel) is the API's job in the next iteration. * docs/www/ai-instruction.md (4a): a table of the BXD files per engine with panel sizes, the UM-HET3 file, "ask the operator for any other panel" with where GN2 records which one is current, and the measured times. Values must be one per individual, in file order -- position, not name -- and the API does not re-index yet, so a mapping is prepared for the caller rather than guessed. * docs/www/operator-checklist.md: MappingMethodId per group, and the three measurements.
11 daysdocs: which genotype file is current, and three real mappings measuredPjotr
The compute endpoints take server-side paths, so callers need to know the deployment's genotype-file layout and which version is current -- neither of which is discoverable from this repository, and both of which are spread across GN2 and GN3 (GN2's per-group genotype/<Group>.json whose `genofile` list is newest-first and is the actual authority -- the GenoFile table still lists only the classic files; GEMMA fed from genotype/bimbam/<file>_geno.txt, R/qtl from genotype/<file>.geno; GN3 reading the sample order out of the .geno header; InbredSet. MappingMethodId deciding which engine a group offers). Measured on this deployment, with the newest genotypes and values re-indexed onto each panel's sample order: BXD BXD.8.geno (WGS Mar2022) GEMMA 17.1s 21055 rows BXD BXD.8.geno R/qtl 7.5s 21056 markers HET3-ITP.12 (Apr2023, 6447 individuals) R/qtl 83.1s 893 markers, lifespan peak LOD 7.95 at 1_24042124 (chr1, 24.04 Mb) That last run is the shape of what Rob Williams wants: UM-HET3 is a 4-way cross (MappingMethodId=3, R/qtl), and its genotype files are versioned per cohort with no JSON recording which is current, and no per-cohort trait split -- 6438 of lifespan's 23879 animal values fall in the .12 panel, the rest are silently dropped. Recorded as the known messiness rather than hidden. * docs/genotypes.md: new. Layout, how GN2/GN3 choose, the alignment rule, per-group engines, the measurements, what is still messy. * README.md: pointer. * docs/www/operator-checklist.md: replace "only the test fixtures exist" with the real panels, and name the missing alignment step.