about summary refs log tree commit diff
path: root/src/genotypes.rs
AgeCommit message (Collapse)Author
4 daysgroups: an endpoint for a family's parental designPjotr
GET /groups/{group}/parents-and-f1s answers the question the mapping notes only implied. It reports the F1 strains, the maternal and paternal strain (from the deployment's parents_and_f1s.json, ported from GN2's ParInfo -- all 24 groups), the @type of the group's genotype file, the individuals each role names where the metadata names them (BXD-Longevity does, most do not), and -- the part a caller needs -- which members of the current panel the default rule drops and which the `F1` name heuristic catches. Two things that came out of building it and comparison with GN2: * the JSON's paternal/maternal labels were INVERTED against GN2's reading. GN2 unpacks ParInfo as `f1, f12, maternal, paternal`, so BXD's maternal strain is C57BL/6J; our file said paternal. No mapping was affected (non_strains takes both regardless of role) but the metadata was wrong, and anything using "maternal" later -- reciprocal F1 direction, the @mat/@pat note -- would have been backwards. The file is rewritten from ParInfo itself (ast-parsed, not retyped), 23 groups' roles filled in and the 24th (BXD-Longevity, which GN2 does not have) left as it was. * an RI set has two kinds of F1 name: the design's (B6D2F1) and the panel's own intercross individuals (C57BL/6JxBXD037F1). The endpoint reports excluded_from_panel and excluded_by_name separately and says which is doing the work, because "0 declared, 37 by name" is otherwise confusing. Empty F1 names (C57BL-6JxC57BL-6NJF2 has two) are dropped rather than becoming exclusion entries that match nothing. genotypes::non_strains is rebuilt on the same typed parser (DesignRole / ParentalDesign), so the rule and the endpoint cannot disagree. tests: a unit test for the roles, empty-name handling and the unlisted-group source, and an integration test asserting BXD's maternal/paternal orientation and the note.
4 daysindividuals: select who enters a mapping, by namePjotr
Parents and F1s are excluded by default (as in GeneNetwork2/3), with include_parents_f1 to keep them all. That boolean was not enough for what was asked -- "include these, drop that substrain" -- and the exclusion list itself is data (parents_and_f1s.json, seven groups) that no request can reach. F1 names also differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH, C57BL/6JxBXD037F1 for the intercross F1s), so pattern matching is the wrong tool. Every group-mapping endpoint -- /compute/gemma, /compute/gemma/ravanan, /compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm -- now takes "individuals": {"include": [...], "exclude": [...]} applied after the parent/F1 rule, exclude winning over include. An unknown name is a 422 invalid_individuals whose message lists the panel: a silently dropped animal changes a kinship matrix without anyone noticing, so the failure has to be loud. genotypes::select_panel is the one implementation, and it returns what it did (dropped parents/F1s, re-included by name, excluded by name) so the Genotype note can say it: "197 individuals, 38 parents/F1s dropped; 1 excluded by name". The note now also spells out the distinction that makes this option treacherous: the kinship uses *every* individual in the genotype file, the association uses only those with a trait value -- GEMMA reads no phenotypes for the kinship. So "include the F1s" changes the relatedness structure without putting the F1s in the fit, and a request can legitimately have 234 individuals in the genotype file and 34 in the association. Verified on BXD.8: excluding BXD1 gives 197 individuals / 33 values; include_parents_f1 with an F1 re-included and a substrain excluded gives 234 / 34. A basket supplies values by name, so a selection applies to a basket-fed mapping the same way: the individuals excluded are simply not in the panel any more, and their basket values are ignored. GenotypeError gains Invalid for this (422) so a caller's typo is not reported as a missing genotype file; the read-only endpoints that can never see one map it the same way for exhaustiveness.
7 daysuploads: every engine maps an upload by name; same name replacesPjotr
Two things asked for, and one bug each. * POST /uploads/genotypes with an existing name now REPLACES it. A genotype is the user's file to keep, not a record the deployment curates; the digest in every reply says which version ran. * genotype_upload on all four engines -- /compute/{rqtl,gemma,qtl2, fitqtl} -- in place of genotype_file/group. The server writes the form each engine reads: a GN .geno for R/qtl and R/qtl2, a BIMBAM pair (new: genoupload::to_bimbam, coding from the base's @mat/@het/@pat, NA for missing, snps as marker/bp/chr) for GEMMA. No user path travels with a request. Verified against the full 3811-marker BXD.4 matrix uploaded as an update: rqtl 3811 rows, gemma 3809, qtl2 3730, fitqtl by marker. The bug worth naming: values given by name were being aligned against a file, and for GEMMA that file is BIMBAM, which has no individual names -- so the phenotype file came out empty and the wrapper failed with a Ruby backtrace that said nothing. GEMMA's values are now ordered by the upload's own individual list before the pair is written. The related fixes: rqtl and fitqtl align named values onto the file's individuals (they previously demanded its column order), and genotypes::align_to_file is the one place that does it. tests/genoupload.rs: mapping an upload on rqtl/qtl2/fitqtl with values by name, and replacing a name (one row in the listing, a new digest). GEMMA's LOCO kinship needs more markers per chromosome than a small fixture has, so it is exercised against a full panel rather than in the fixture.
8 daysgenotypes: call frequencies, marker names, and a location per markerPjotr
Two questions the genotype surface could not answer: what a panel is actually called as, and which markers it holds. The listing gives counts and the compute endpoints give scans; a panel whose calls are not the codes its header declares, or a marker that is 99% missing, stayed invisible until a scan came back flat -- and then the search for the cause started at the wrong end. * GET /groups/{group}/genotypes/{version}/frequencies counts the calls in the version's .geno and returns them as call -> frequency ({"B": 0.4889, "D": 0.492109, "H": 0.018991}). Frequencies are over the calls that are present, so they sum to 1; missing codes (U, N, NA, an empty cell) are counted separately and reported in notes, because the missing rate is the diagnosis. * A probability-valued file is binned 0.05 wide (twenty bins over [0, 1], keys are the bin edges), since 0.4187 and 0.4191 are the same statement about a marker; ?bin= sets another width. A file with a few distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported exactly -- binning three values hides what is being looked at. * .../markers/{marker}/frequencies is one locus, and ?marker= on the panel route is the same call. Counting goes through an interner, so cost is one hash lookup per cell and one allocation per distinct call; a .geno over 256 MB is refused (413 genotype_too_large) rather than read, which is what HSNIH-Palmer_true (3.7 GB, 134927 markers x 6147 individuals) demanded. * The version's /markers now lists marker NAMES only, one column, and .../markers/{marker}/location returns {"name","chr","cM","Mb"} as named fields: a page of names stays small, and a position is a lookup rather than a table the client must hold. The same size guard applies, so an oversized panel's listing is a 413 in 3 ms. * Files no group lists -- a panel in preparation, a fixture, the DO's do/ cross -- live at /groups/unlisted/genotypes/{name}/... with the same four calls. `unlisted` is a namespace, not a group: only its genotype subtree answers. One prefix for the whole genotype surface beats a second spelling for the same tally, so the earlier /genotypes/{name} spelling is gone. * genotypes::resolve_named (moved out of the frequencies module, since it is genotype metadata) resolves a version, a .geno name, or a path relative to the genotype files directory; MAX_SCAN_BYTES is shared by every whole-file read. Tests limited to the feature: genotypes 3 (names-only columns, paging, location fields, marker_not_found, unlisted parity, 413), frequencies 5, frequencies unit 7.
8 dayscompute: R/qtl2 as a third scan, POST /compute/qtl2Pjotr
R/qtl sits between the existing two scans: Haley-Knott on marker genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with mean-imputed genotypes. Neither can analyse a multiparent population, and neither models relatedness from the actual genotype data. R/qtl2 does both -- HMM genotype probabilities, then an LMM with leave-one-chromosome-out kinship -- and it is what names the third peak on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24, against GEMMA's -log10 p 7.15 there). * src/qtl2.rs converts this deployment's own .geno into a qtl2 cross under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with the genotype-code map, the orientation and the paths written out explicitly. The shipped genotype/rqtl2/ control files cannot be used -- they point outside the data directory, declare no code map (every genotype reads as missing) and no orientation (individuals read as markers) -- and each of those produces a plausible flat scan instead of an error, which is why the conversion is ours. * Heterozygous calls are dropped on a riself panel, as R/qtl's convert2riself does. Left in, qtl2 reads the cross as multiparent and its scan1 takes a path the panel cannot satisfy ("Error in scan1_pg ... : No individuals in common"). * Autosomes only: X needs a sex covariate to be fitted properly and Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is restricted the same way. The note says how many markers were left out. * scripts/qtl2_scan.R caches genotype probabilities and kinship by a hash of the control file *and its directory*; two conversions of the same panel can have identical control files and identical file sizes, and reusing the first one's probabilities silently scans the wrong cross. It also takes the individual names from whichever shape the cross's pheno slot has -- a single-column pheno file leaves it a plain matrix, and the resulting empty phenotype reads as scan1's "No individuals in common". * Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply unchanged -- post the same samples as /compute/rqtl. * tests/qtl2.rs: the fixture scans and its notes name the cross; BXD trait 10001 peaks on chr8 under both scans; refusals carry missing_genotype / invalid_kinship. cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like R/qtl; a DO cross costs minutes of genotype probabilities, paid once per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
9 daysfix: DOL maps again on both engines, and the four stale testsPjotr
DOL (200 individuals, 112 913 markers, one genotype version) failed on R/qtl with "There are missing chromosome IDs" and on GEMMA with a bare wrapper bail-out. Two unrelated causes, both in our own code. **R/qtl: a genotype code collided with a chromosome name.** DOL's header says `@unk:X` -- X is the code for a missing genotype -- and the wrapper passed that to `read.csv(na.strings=...)` for the *whole* file, so every marker on the X chromosome lost its chromosome ID and `read.cross.csv` refused the map. The unknown code now applies to the genotype columns only. DOL scans: 112 913 markers in 42 s, where before it never got past reading the file. Same fix in scripts/qtl_fitqtl.R, which shares the read. **GEMMA: its leave-one-chromosome-out run cannot fit the Y.** With LOCO the DOL panel is scanned per chromosome, and the Y run dies inside BLAS -- `Enforce failed for N>0 in src/fastblas.cpp ... fast_cblas_dgemm` -- taking the whole request with it (a 502 whose detail is the wrapper's banner). The mitochondrial chromosome has the same shape of problem. `prepare` for the GEMMA engine now writes the bimbam pair without the Y and M markers (cached beside the individual subset, line-aligned so the geno and snps stay in step), and the note says what was left out: "46 Y/M markers left out (GEMMA's LOCO cannot fit them)". DOL maps: 106 515 rows in 49 s. **And the four tests that had been failing since before this work**, all stale expectations rather than product bugs: * `benchmark_questions` pinned two traits to "page 2" of a 100-per-page search when they rank ~190th and ~219th of 246 -- one re-index pushes them over. It now walks up to three pages and asserts presence and LOD, not the page; and the helper `search()` pages, because a benchmark row can name a trait that ranks outside the first hundred of its own query. * `no_db_mode_degrades_gracefully` asserted exactly one hit for "alcohol" from the index alone. The index grew; the behaviour under test is that the search still answers, within the limit, with the database-backed fields null. That is what it asserts now. * `search_proxy_scenarios` and `search_parity_replay` never pinned the engine they test, so an ambient `SEARCH_ENGINE=native` made the proxy scenarios answer from the local index: a 200 where a 422 was expected, and "expected 2 upstream calls, got 0" for every replayed query. Both now set `SEARCH_ENGINE=forward`. * src/genotypes.rs: `DEGENERATE_CHROMOSOMES`, `drop_degenerate_chromosomes`, `Prepared.markers_dropped`. * src/routes/gemma.rs: also the 500 that `eb5320d` introduced -- counting parents/F1s from `panel - keep` instead of subtracting the exclusion *set*, whose length can exceed the panel (subtract overflow). * scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: na.strings scoped to the genotype columns. * tests/{benchmark,no_db,search,search_parity}.rs: as above. Verified through the API on DOL trait 10003: R/qtl 112 913 rows, GEMMA 106 515 rows. gemma/rqtl/fitqtl suites green (14, 10, 4).
9 daysgemma: leave out individuals with no genotypes, which make the kinship singularPjotr
Asking whether the singular matrices could be fixed turned out to have two answers, one of them a real bug on our side. The bug: a panel can name individuals whose genotype row is empty, and an empty row makes the kinship matrix singular. GEMMA then reports "eigenvalues close to zero" and dies in LAPACK with `LUDecomp ... input domain error` -- a message about linear algebra for a problem that is a missing genotype. Measured on this deployment's own files: BXD-AE.4 136 of 344 individuals have no genotypes MDP 93 of 178 BXD-Heart-Metals.4 200 of 481 HET3-ITP.3 2867 of 2867 (the whole file is empty) Dropping them makes those panels map: BXD-AE.4's GWA step goes from a failure to 3.2 s, and MDP's kinship step from a failure to success. The individuals contribute nothing to a scan either way, so nothing is lost; `notes` counts them, and the "Genotype:" line no longer blames parents/F1s for them (`prepared.excluded` is "panel minus kept", which now includes both, so the two counts are separated). Not the cause, but worth naming: `AXB` fails earlier still, on its annotation -- `ERROR: Enforce failed for D1Mit296 in src/gemma_io.cpp ... ReadFile_anno` -- which is a broken `.snps` file, not a matrix. That wants a pre-flight check of the annotation (report the offending marker) rather than a GEMMA failure, and is not in this commit. Considered and rejected: a ridge on K (`K + epsilon*I`) would work mathematically but GEMMA has no option for it and the wrapper caches K by content hash, so it would mean patching the wrapper's cached matrix; `-gk 2` and dropping markers do not add rank; and the non-LOCO path is the one that does not run at all here. * src/genotypes.rs: `individuals_with_genotypes`, one pass over the bimbam file marking the individuals with any genotype at all. * src/routes/gemma.rs: filter `keep` by it before alignment (so the trait values drop with them), and say so in `notes`. * tests/gemma.rs: `individuals_with_no_genotypes_are_left_out` -- BXD-AE.4 maps, the note names the drop and the reason, and the genotype line no longer claims 344 individuals. * README.md: the behaviour and why it exists, under /compute/gemma. 307 tests pass; the same four as before do not.
11 daysgenotypes: expose the metadata, never the genotype callsPjotr
The listing named versions but not files; a caller could pick one and still not see what was in it. Three token-gated routes now describe a version completely, short of the matrix: GET /groups/{group}/genotypes versions, files, sizes, current GET /groups/{group}/genotypes/{version} file name, header labels, individuals, strains, chromosomes GET /groups/{group}/genotypes/{version}/markers markers (name, chr, cM, Mb), paged The detail reads the file's own `@`-labels (type, mat, pat, het, unk, scale) and its panel in column order, and reports both the whole panel and the strains a mapping uses after the parents/F1s rule -- so a caller can check a request before sending it and interpret a result after. Markers come back as positional rows with `meta.columns`, paged with `?page`/`?limit` and filterable with `?chr`, following /compute/gemma's shape rather than inventing one. The genotype calls stay on the server: file_meta() reads the matrix to skip it, and nothing returns a cell. * src/genotypes.rs (Marker, FileMeta, file_meta; GenotypeFile::file): header labels, individuals and marker positions; unit test. * src/routes/genotypes.rs: the two new handlers, token-gated like the listing, plus `file` in the listing and counts in the detail. * tests/genotypes.rs: the listing carries file names but no paths. * README.md, docs/genotypes.md, docs/www/ai-instruction.md: what the three routes give, and that the matrix is not among them.
11 daysgenotypes: list a group's genotype files, and map by group + versionPjotr
Until now /compute/{rqtl,gemma} took server paths, so the caller had to know the deployment's layout, pick a version, and expand a trait's values onto the panel in the file's order -- none of which the API told it. The listing endpoint and the group-scoped request remove all three: GET /groups/{group}/genotypes versions, newest first, no paths POST /compute/rqtl {group, [genotype_version], samples} POST /compute/gemma {group, [genotype_version], samples} The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the group's genotype/<Group>.json `genofile` list is newest-first and its first entry is current; groups with no JSON (HET3-ITP) fall back to the highest version number, and the listing's notes say which happened -- reads the panel from the .geno header the way GN3 does, and excludes parents and F1s by default (GN2's `incparentsf1` off, GN3's `get_sample_list(inc_par=False, inc_f1=False)`). Excluding them has to subset the genotype matrix, not null-fill the values: a null keeps an individual out of the fit but not out of GEMMA's kinship or R/qtl's cross. So `prepare` writes a panel-subset file under the engine's cache directory, keyed by panel, and reuses it. Nothing in the listing or the mapping response carries a path: the endpoint reports versions and sizes, and `notes` names the resolved version, the panel size and how many values aligned. Explicit `genotype_file`/`snps_file` remain as an override for files outside the layout, and the two request shapes are mutually exclusive (422). * src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1 exclusion, subset preparation, name alignment; 10 unit tests. * src/routes/genotypes.rs: new endpoint, with a test that its list has no path or file name in it. * src/routes/{rqtl,gemma}.rs: group + genotype_version + include_parents_f1; named `samples` aligned server-side; notes record what resolved. gemma also accepts named samples where it used to insist on positional trait_values. * src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now carry `group` and no placeholders; the dataset-scoped ones keep the placeholder and say to add the group. * src/routes/groups.rs: the group record points at /genotypes. * src/config.rs: GENOTYPE_FILES (default the gn4-data directory). * tests/genotypes.rs: new integration test (skips without the directory). * README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the versions, map by group" -- measured runs, the parent/F1 rule, the subsetting requirement, and the restart list unchanged.