about summary refs log tree commit diff
path: root/src
AgeCommit message (Collapse)Author
8 daysuploads: every engine maps an upload by name; same name replacesPjotr
Two things asked for, and one bug each. * POST /uploads/genotypes with an existing name now REPLACES it. A genotype is the user's file to keep, not a record the deployment curates; the digest in every reply says which version ran. * genotype_upload on all four engines -- /compute/{rqtl,gemma,qtl2, fitqtl} -- in place of genotype_file/group. The server writes the form each engine reads: a GN .geno for R/qtl and R/qtl2, a BIMBAM pair (new: genoupload::to_bimbam, coding from the base's @mat/@het/@pat, NA for missing, snps as marker/bp/chr) for GEMMA. No user path travels with a request. Verified against the full 3811-marker BXD.4 matrix uploaded as an update: rqtl 3811 rows, gemma 3809, qtl2 3730, fitqtl by marker. The bug worth naming: values given by name were being aligned against a file, and for GEMMA that file is BIMBAM, which has no individual names -- so the phenotype file came out empty and the wrapper failed with a Ruby backtrace that said nothing. GEMMA's values are now ordered by the upload's own individual list before the pair is written. The related fixes: rqtl and fitqtl align named values onto the file's individuals (they previously demanded its column order), and genotypes::align_to_file is the one place that does it. tests/genoupload.rs: mapping an upload on rqtl/qtl2/fitqtl with values by name, and replacing a name (one row in the listing, a new digest). GEMMA's LOCO kinship needs more markers per chromosome than a small fixture has, so it is exercised against a full panel rather than in the fixture.
8 daysuploads: map an uploaded genotype by name, and clean up after itPjotr
Three follow-ups from putting an upload through the whole loop. * /compute/qtl2's `genotype_file` branch only accepted positional `trait_values`, i.e. it demanded the file's column order -- which a caller with an uploaded population does not have. It now takes `samples` (name and value) or a `basket` and aligns them onto the cross by name, leaving individuals without a value as missing. Positional `trait_values` still work for a file whose order is known. * DELETE /uploads/genotypes/{name} now removes the .geno it wrote on demand: a file nobody can name is a file that confuses the next operator. * scripts/example-genotype-upload.py spreads its markers across chromosomes. One chromosome of markers is a poor example -- a leave-one-chromosome-out kinship has nothing to leave out, so the scan came back with no rows at all, which looked like a broken upload and was really a property of the example. 20 markers over the genome map cleanly (19 rows, the 20th used for the HMM), and the README says to use "kinship": "overall" if a file really has one chromosome.
8 daysuploads: a genotype matrix that updates an existing populationPjotr
Tab-delimited, header names the individuals from column 4, then marker, chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own SQLite file -- nothing goes near MariaDB -- and written out on demand as a GN .geno carrying the base's @-metadata, so the R scripts read exactly what was uploaded. An upload updates a population, it does not invent one, and the checks say so: * the base version must exist (resolved through the same genotypes::resolve_named the rest of the API uses); * at most 50 individuals the base does not have, and at least one that it does -- no overlap is a different population, not an update; * at most 200 000 markers, and the refusal says the sysadmin has to organize a file that size, because this is a request path, not an ingest; * the values must be the same kind as the base's: letters, whole numbers or probabilities. A letters panel holding 0/1 would be read as the wrong thing silently, so it is refused naming both kinds. GET /uploads/genotypes lists them, /{name} its metadata (overlap, new individuals, kind, digest), /{name}/markers pages the marker table, and /{name}/geno writes the file and returns the path for `genotype_file` on the compute endpoints. DELETE removes one. scripts/example-genotype-upload.py turns any .geno into a valid update -- a subset of markers, a few individuals dropped and added -- and tests/genoupload.rs builds its own from the deployment's BXD.4 for the same reason: upload, page the markers, write the .geno and check the layout, delete, then break each rule (no overlap, numbers in a letters panel, commas) and check the refusal says which. Unit tests cover the three value kinds, missing cells, ragged rows, a non-numeric position, the 50-new-individual limit, and the .geno layout (Chr first, then the marker -- GN's order, not the upload's).
8 daysbaskets: clear orphan rows on open, so an affected file healsPjotr
The cascade fix cleans files written from now on, but a file written before it still carries rows whose basket was deleted without them -- the production file did, and the reused rowid made the next create fail with UNIQUE constraint failed. Two indexed deletes on open make such a file heal instead of failing forever, and a test simulates the old bug by deleting a basket row by hand and then creating another.
8 daysmatrices: upload new traits and covariates as one named tablePjotr
The other half of changing data: bringing new columns in. A matrix is a delimited table -- a header row, then one row per individual, the first column naming them -- stored in the uploader's own SQLite file and checked against an existing genotype version's panel before it is accepted. * POST /matrices {name, genotype_version, contents, kinds?}: parse the table (tab, comma or semicolon, guessed from the header), then check it against that version's individuals. The count must match and every name must be in the panel; 92 of 93, or a name the panel does not have, is a 422 that says which. A mapping that silently drops individuals is worse than an error. * kinds marks columns that are covariates; everything else is a trait. * Each column becomes a basket named <matrix>.<column> -- a dot, not a slash, because a basket name has to survive a URL path segment -- so a column maps on any engine with {"basket": "my-traits.bw"} and inherits the revision history and digest without a line of new mapping code. * GET /matrices lists them, GET /matrices/{name} their columns and basket names, GET /matrices/{name}/values the table back (parsed values, null where missing), DELETE removes the matrix and its columns. Two bugs this found in the basket store, both real: * SQLite has foreign keys OFF by default, so the ON DELETE CASCADE never fired: deleting a basket left its rows behind, rowids were reused, and the next create hit "UNIQUE constraint failed: basket_version.basket_id, revision". The pragma is now set, and deletion removes the dependants explicitly, because files written before it have orphans. * The first attempt named a column's basket "<matrix>/<column>", which the basket-name rule (rightly) refuses. The dot keeps both rules. scripts/example-matrix.py writes a table from a genotype file's own header with deterministic values, so the fixture passes the panel check by construction -- tests/matrices.rs builds its own for the same reason and walks the loop: upload, read back, map a column with R/qtl2, delete, plus the three refusals (too few individuals, a name not in the panel, an unknown version).
8 daysbaskets: every revision kept, files named for the user, all four enginesPjotr
Three things the first cut was missing, and both were asked for. * An automatic version history. Revision 1 is the column as first stored and each PATCH adds one -- with its own digest and Unix timestamp -- and earlier revisions are kept. GET /baskets/{name}/revisions is the history, ?revision=N reads an old column, and a mapping can pin one: {"basket": "wbc-edit@2"}. That is the point of a digest: a result quoted yesterday can still be reproduced after the column has moved on. * Files named for the user, not a hash of the subject: BASKET_DIR/<user>.sqlite (0600, WAL). Tokens rotate, subjects do not, so a new token finds the same baskets -- and an operator can see whose baskets a file holds. A subject that is not filename-safe is sanitised and disambiguated with a short digest, and a pre-naming hash file is renamed on first use rather than orphaned. * /compute/rqtl and /compute/gemma and /compute/fitqtl take `basket` too, so all four engines share one meaning: a basket resolves to the same (name, value) pairs a client would have posted, and every reply records the revision and digest it ran. The schema upgrade is a real one, not a reset: an older file's single column becomes revision 1 of the history, cells copied across with the digest and timestamp they already had. Tested by building the old schema by hand and opening it (an_older_file_is_upgraded_with_its_column_intact), which is also how I found that the migration was recreating a table the base schema had already made. Verified on all four engines, head and pinned: rqtl 3811 rows, gemma 3809, qtl2 3730, fitqtl 3 markers -- each reply carrying "Basket '...' revision N (digest ...)".
8 daysbaskets: a user's own column of values, in their own SQLite filePjotr
The next phase is changing data, and the constraint is that MariaDB is never written to. A basket is the storage half of that: take a trait's values, edit some of them (or build a covariate column that does not exist), keep the column under a name, and map that instead. * POST /baskets stores a complete column; GET /baskets lists yours; GET /baskets/{name} its metadata; GET /baskets/{name}/values the cells; PATCH /baskets/{name} replaces the column with a new revision; DELETE removes it. * POST /compute/qtl2 takes "basket": "<name>" in place of `samples`, and both at once is a 422 rather than a silent choice. The mapping code below it is untouched -- a basket resolves to the same (name, value) pairs a client would have posted. * One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600, WAL, busy_timeout. `GET /baskets` only ever lists the caller's own; the file name is a hash, so a directory listing does not name users. * The full column is stored, not a diff -- the caller has the original beside it, and a mapping needs one complete column. * Text with a parsed copy beside it: raw (what was typed), value (the number or NULL), status (ok/missing/bad). "NA" and 0 stay different claims, and a column with something that is neither a number nor a missing marker (NA, x, -, ., ?) is refused with the samples named -- silently dropping it would change a trait without saying so. * A revision and a digest: an edit bumps the revision and changes the digest, and a mapping reply records both ("Basket 'wbc-edit' revision 2 (digest 9559a70e...)"), so two runs on one name cannot look alike. A result cache must key on the digest, not the name. The loop, end to end on BXD trait 10001: 34 values posted as a basket, one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2 -- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the revision and digest. tests/baskets.rs walks that loop (create, list, values, patch, map by name, delete) and the refusals: a non-number named in the error, a name that cannot become a path, an unknown basket, and basket-plus-samples. Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest changing on edit, per-user isolation, and name validation. rqtl/gemma take the same 10-line change next -- their `samples` shapes differ, so they are a separate commit rather than a rushed one.
8 daysapi: scans point at the class counts behind their own peakPjotr
The gotcha notes in SKILL.md tell an agent to count the animals per genotype class before believing a peak; a note in a document is only found by whoever reads it, so the replies now say it too, at the peak they just computed. * suggest::class_hint builds /groups/{g}/genotypes/{v}/markers/{peak}/classes?trait={trait_id} -- a real GET with one placeholder, because a scan knows the trait's values, not its PublishXRef.Id. * /compute/{rqtl,gemma,qtl2,fitqtl} add it to `next` after the PheWAS link, naming the peak each reply found and the version the request named (a {version} placeholder when it named none). * The genotype subtree carries the template too: a version's detail, and a concrete marker's own links. * /compute/qtl2 gains the sentence that explains the divergence: on a riself (riset) panel heterozygous calls are dropped before the scan (R/qtl's convert2riself) and the HMM re-imputes them, while GEMMA keeps them at dosage 0.5, so a mostly-H marker cannot peak there and two engines can name different markers of one region without either being wrong. tests/links.rs: a scan's reply offers the class counts at its own peak -- the link names the version and the peak marker the reply reports, and leaves {trait_id} for the caller.
9 daysgenotypes: a trait summarised per genotype class at one markerPjotr
The question that follows "why do GEMMA and R/qtl2 peak at different markers": what does the trait look like within each genotype class at one marker. The frequencies endpoints report the distribution of calls, not the values behind them, and the answer must not be "serve the matrix" -- this API never does. GET /groups/{group}/genotypes/{version}/markers/{marker}/classes?trait=<id> -> {"A": {"n", "mean", "sd", "min", "max"}, "H": {...}, "B": {...}} The calls are joined to the trait's per-strain values inside the server (the same lookup, authorization and 404s as /groups/{g}/traits/{id}/values) and reduced to those five numbers per class. Nothing in the payload names an animal. A class with fewer than ten animals is called out in notes, because a peak resting on two is a different claim from one resting on a hundred. It settles the DOL trait 10003 question. At GEMMA's top marker JAX00418132 (chr16 36.35 Mb) the classes are A n=2 mean 1.8, H n=20 mean 2.53, B n=171 mean 3.95 -- the A class is two animals and they are the trait's two lowest values. R/qtl2 drops H on a riself panel (R/qtl's convert2riself), so it keeps 173 animals there but only 2 of one class: nearly monomorphic, which is why its scan peaks elsewhere. GEMMA keeps the 20 hets at dosage 0.5, and that is where its sharper p-value comes from. GEMMA's next two hits are the same pattern in opposite directions (one A among 171, then two B among 171), so the chr16 block is a haplotype carried by a handful of extreme animals, not a population effect. Also fixes the reading of GEMMA's `af`: it is mean(dosage)/2 under the BIMBAM coding this deployment ships (B=1, H=0.5, A=0), so 0.469 for JAX00418132 is half the B-allele frequency 0.935, not the A-allele frequency 0.065. The bimbam file and DOL.geno agree at that marker. tests/frequencies.rs gains a route test: the class fields, A < B, the small-class note, the "not served" note, no individual names anywhere in the payload, and a 404 for a trait the group does not have.
9 daysgenes: give the Wikidata call 30 s, not 20Pjotr
Two SPARQL queries in a row, and Wikidata's query service measured 9 s for one of them during the deploy -- so the first call for a symbol timed out at 20 s and answered 502 while the service was merely slow. Only the first call pays it; the answer is cached for a week. The refusal now says how long it waited.
9 daysgenes: Wikidata aliases, cached in gene-aliases.lmdb for seven daysPjotr
A GeneNetwork trait page shows the gene's aliases next to the symbol, and GN4 had no way to ask for them: gn-guile has a Wikidata lookup (gn/db/sources/wikidata.scm) and nothing in GN4 did. This is that lookup with the caching Wikidata deserves -- it is a shared public service and an alias list is not news. * GET /genes/{symbol}/aliases: two SPARQL queries against WIKIDATA_ENDPOINT (the entity: a gene wdt:P31 wd:Q7187 in human, mouse or rat, labelled with the symbol; then its skos:altLabel s in English), cached in WIKIDATA_CACHE_DIR/gene-aliases.lmdb, an LMDB file (NO_SUB_DIR, 0600, so one file and LMDB's -lock companion rather than a directory of two). * TTL WIKIDATA_TTL_SECONDS, seven days by default. Past it the value is refetched; ?refresh=true forces that. * The label is matched case-insensitively, which the gn-guile query did not: human genes are labelled SHH and mouse ones Shh, so an exact match finds only one species. ?species=human now returns the ortholog's labels -- HLP3, HPE3 and the rest -- which are the ones a mouse trait page shows. With no species named the pick is deterministic (mouse, rat, human), so a repeated request cannot change entity. * A failed refresh keeps the last value: the answer says stale: true, the envelope is partial, and a note explains it. Nothing cached at all is a 502 that says so. An empty alias list and an unreachable upstream are different facts, and only one of them is about the gene. Seven unit tests (key scoping, TTL including a clock that went backwards, the query shape and quote-escaping, SPARQL flattening, an LMDB round-trip that asserts the file is gene-aliases.lmdb, and stale-on-failure with and without a cache) and two route tests (the second call is served from the cache -- same fetched_at -- and the species switch), which skip with a 502 assertion when there is no network.
9 daysprobesets: cross-dataset lookups by probeset id and by gene symbolPjotr
A probeset id is not a Xapian field, so /search cannot find 1427571_at exactly and the only route to it was guessing a dataset and calling /datasets/{ds}/probesets/{name} until one answered. A gene symbol could be searched, but the reply was capped at 100 hits, interleaved probesets and platforms, and carried no total -- so "did I see all of Shh?" had no answer. Both questions are one indexed query each: ProbeSet.Name_IDX and symbol_IDX. * GET /probesets/{name}: one row per dataset that carries the probeset, with the panel (group, species, platform) and the scan's statistics (mean, LRS, LOD, locus, additive). Suffixed chip variants match too -- 1427571_at also finds 1427571_at_A -- and each row's `name` says which matched, so the answer never hides a substitution. LIKE's wildcards are escaped, or every Affymetrix `_at` would match any character. 155 rows over 155 datasets in 32 ms. * GET /genes/{symbol}/traits: the same rows filtered on Symbol, with a real total, ?sort=lod, ?min_lod=, ?group=, ?species=, ?platform= and paging -- "BXD datasets where Shh is a strong eQTL" is one call. 1225 rows over 17 datasets and 13 probesets in 38 ms. * Both serve metadata, not values. A token is enough for the index; trait values stay per-dataset and authorized per dataset, and `next` goes to /datasets/{ds}/probesets/{name}/values, which checks. There is no cross-dataset values endpoint, deliberately: that is bulk work and /groups/{g}/traits is the bulk call. The one-statement join took 5.2 s for Shh -- MySQL rescans InbredSet/Species per row -- so the 933 ProbeSetFreeze rows and the 102 panels are fetched separately and joined in memory: 70 ms each. nsamples is left out: it is a ProbeSetData count per row (+0.5 s) and the dataset record already reports strains.
9 daysapi: every response suggests where to go nextPjotr
A caller that knows only /version should be able to reach anything by following links. Until now `next` was thin and uneven: the genotype subtree offered none at all, the group record offered one, /version offered nothing, and two links it did offer could not be followed. * suggest:: now builds the graph: root_links (groups, species, mapping-methods, openapi), genotype_version_links (frequencies, marker names, a marker's location and tally, the group, the scanning actions), unlisted_genotype_links, marker_links (location, tally, marker list, panel tally, PheWAS at that marker), group_links and trait_links. Envelope::with_links / with_links_except add them, deduplicating and never suggesting the response the caller is already reading. * Wired through the genotype surface (listing, version detail, markers, location), both frequencies routes, the compute endpoints, groups, traits, both value routes, species, datasets, publications, accessions, search, mapping methods, phewas, correlation and /version. * Fixed what was not followable, because `next` promises a GET that returns 200 and a client is the one that finds out otherwise: - /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are gone; comput endpoints now link ?locus=<their strongest marker> (suggest::peak_marker_of, computed before the rows move). - /datasets and /publications are not routes; root_links offers the listings that exist (/groups, /species, /mapping-methods) and /openapi.json. - /search?q= with an empty query is a 400, so it is not a link. - the correlation endpoint's "/compute/correlation" was a POST in `next`; it is in `actions` now. - marker_links pointed at /markers/{marker}, which is not a route; it points at /markers/{marker}/location. tests/links.rs walks the graph two hops from a dozen seeds and fails on any link that does not answer 200, plus a check that a tally leads to the marker list and to a marker's location, and a location to that marker's tally and its PheWAS.
9 daysgenotypes: call frequencies, marker names, and a location per markerPjotr
Two questions the genotype surface could not answer: what a panel is actually called as, and which markers it holds. The listing gives counts and the compute endpoints give scans; a panel whose calls are not the codes its header declares, or a marker that is 99% missing, stayed invisible until a scan came back flat -- and then the search for the cause started at the wrong end. * GET /groups/{group}/genotypes/{version}/frequencies counts the calls in the version's .geno and returns them as call -> frequency ({"B": 0.4889, "D": 0.492109, "H": 0.018991}). Frequencies are over the calls that are present, so they sum to 1; missing codes (U, N, NA, an empty cell) are counted separately and reported in notes, because the missing rate is the diagnosis. * A probability-valued file is binned 0.05 wide (twenty bins over [0, 1], keys are the bin edges), since 0.4187 and 0.4191 are the same statement about a marker; ?bin= sets another width. A file with a few distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported exactly -- binning three values hides what is being looked at. * .../markers/{marker}/frequencies is one locus, and ?marker= on the panel route is the same call. Counting goes through an interner, so cost is one hash lookup per cell and one allocation per distinct call; a .geno over 256 MB is refused (413 genotype_too_large) rather than read, which is what HSNIH-Palmer_true (3.7 GB, 134927 markers x 6147 individuals) demanded. * The version's /markers now lists marker NAMES only, one column, and .../markers/{marker}/location returns {"name","chr","cM","Mb"} as named fields: a page of names stays small, and a position is a lookup rather than a table the client must hold. The same size guard applies, so an oversized panel's listing is a 413 in 3 ms. * Files no group lists -- a panel in preparation, a fixture, the DO's do/ cross -- live at /groups/unlisted/genotypes/{name}/... with the same four calls. `unlisted` is a namespace, not a group: only its genotype subtree answers. One prefix for the whole genotype surface beats a second spelling for the same tally, so the earlier /genotypes/{name} spelling is gone. * genotypes::resolve_named (moved out of the frequencies module, since it is genotype metadata) resolves a version, a .geno name, or a path relative to the genotype files directory; MAX_SCAN_BYTES is shared by every whole-file read. Tests limited to the feature: genotypes 3 (names-only columns, paging, location fields, marker_not_found, unlisted parity, 413), frequencies 5, frequencies unit 7.
9 dayscompute/qtl2: multiparent crosses, by namePjotr
A multiparent population cannot come from a .geno. That file carries two codes per marker (B/D, A/H/B) -- a collapse, not a founder allele -- and the collapse is not invertible, so DOL's 194 DO animals cannot be fitted as a DO however much reference data appears. What can be is a cross that ships its own qtl2 control file with founder genotypes, so the endpoint now takes one by name: {"rqtl2_cross": "do", "trait_name": "DO::10001 (WBC)", "samples": ...} * resolve_control_file looks in genotype/rqtl2/{name}.json, then {name}/{name}.json (do/do.json), then {name}.json under GENOTYPE_FILES; prepare_control_cross reads the cross's pheno, maps and founder_geno headers without converting anything. It reports crosstype, founders and whether the cross is multiparent, and refuses a multiparent crosstype with no founder_geno rather than silently scanning it as biallelic. * Samples are matched to the cross's own population by name and the strays are counted in notes; rqtl2_cross cannot be combined with group or genotype_file. * The gate is given individuals x founders, because a multiparent scan fits one coefficient per founder allele at every marker. * Autosomes only now applies to shipped crosses as well: the DO's X is left out (369 markers) instead of being fitted without a sex covariate, and the script subsets with cross[, chr = autosomal]. Measured on the DO (WBC, 742 individuals, 8 founders, 6044 autosomal markers): probabilities 174 s / 1.39 GB (cached), LOCO kinship 15 s (cached), scan 21 s. scan1 runs on one core for a multiparent cross because its forked path stalls on allele probabilities here -- 25+ minutes at 8 cores with no CPU consumed, against 21 s serial. Two bugs found by running an actual published trait through it: * do_pmap.csv is in Mb, not the bp qtl2 documents, so every DO peak was reported at 0.0 Mb. The map's unit is now detected from its largest position and printed. * The phenotype file was read with the default whitespace separator, so a label with a space ("DO::10001 (WBC)") split into three fields and read.table stopped with "line 1 did not have 3 elements". Both the pheno and covariate reads are tab-separated now. The R error extraction also joins R's wrapped continuation line, so a complaint reads as a sentence rather than "Error in scan(...) :".
9 dayscompute: R/qtl2 as a third scan, POST /compute/qtl2Pjotr
R/qtl sits between the existing two scans: Haley-Knott on marker genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with mean-imputed genotypes. Neither can analyse a multiparent population, and neither models relatedness from the actual genotype data. R/qtl2 does both -- HMM genotype probabilities, then an LMM with leave-one-chromosome-out kinship -- and it is what names the third peak on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24, against GEMMA's -log10 p 7.15 there). * src/qtl2.rs converts this deployment's own .geno into a qtl2 cross under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with the genotype-code map, the orientation and the paths written out explicitly. The shipped genotype/rqtl2/ control files cannot be used -- they point outside the data directory, declare no code map (every genotype reads as missing) and no orientation (individuals read as markers) -- and each of those produces a plausible flat scan instead of an error, which is why the conversion is ours. * Heterozygous calls are dropped on a riself panel, as R/qtl's convert2riself does. Left in, qtl2 reads the cross as multiparent and its scan1 takes a path the panel cannot satisfy ("Error in scan1_pg ... : No individuals in common"). * Autosomes only: X needs a sex covariate to be fitted properly and Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is restricted the same way. The note says how many markers were left out. * scripts/qtl2_scan.R caches genotype probabilities and kinship by a hash of the control file *and its directory*; two conversions of the same panel can have identical control files and identical file sizes, and reusing the first one's probabilities silently scans the wrong cross. It also takes the individual names from whichever shape the cross's pheno slot has -- a single-column pheno file leaves it a plain matrix, and the resulting empty phenotype reads as scan1's "No individuals in common". * Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply unchanged -- post the same samples as /compute/rqtl. * tests/qtl2.rs: the fixture scans and its notes name the cross; BXD trait 10001 peaks on chr8 under both scans; refusals carry missing_genotype / invalid_kinship. cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like R/qtl; a DO cross costs minutes of genotype probabilities, paid once per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
9 daysgemma: quote GEMMA's own error, not the wrapper's bannerPjotr
Every GEMMA failure the sweep produced read the same way -- "gemma-wrapper bailed out with pid ... exit 1", sometimes with the Ruby backtrace -- which says nothing about whether the API or the panel is at fault. The line that does is GEMMA's, printed in the middle of the wrapper's per-chromosome chatter: ERROR: Enforce failed for N>0 in src/fastblas.cpp at line 194 in fast_cblas_dgemm so `CommandFailed` now carries the last GEMMA diagnostic (an `ERROR`, `error!`, `****` warning, `Enforce failed` or "not enough" line) rather than the raw stderr, falling back to the wrapper's own tail when GEMMA said nothing at all. The wrapper's banner and `.rb:` backtrace lines are skipped: they are the noise this exists to get past. Two tests: the complaint is GEMMA's line, not the banner; and a silent failure still names something.
9 daysfix: DOL maps again on both engines, and the four stale testsPjotr
DOL (200 individuals, 112 913 markers, one genotype version) failed on R/qtl with "There are missing chromosome IDs" and on GEMMA with a bare wrapper bail-out. Two unrelated causes, both in our own code. **R/qtl: a genotype code collided with a chromosome name.** DOL's header says `@unk:X` -- X is the code for a missing genotype -- and the wrapper passed that to `read.csv(na.strings=...)` for the *whole* file, so every marker on the X chromosome lost its chromosome ID and `read.cross.csv` refused the map. The unknown code now applies to the genotype columns only. DOL scans: 112 913 markers in 42 s, where before it never got past reading the file. Same fix in scripts/qtl_fitqtl.R, which shares the read. **GEMMA: its leave-one-chromosome-out run cannot fit the Y.** With LOCO the DOL panel is scanned per chromosome, and the Y run dies inside BLAS -- `Enforce failed for N>0 in src/fastblas.cpp ... fast_cblas_dgemm` -- taking the whole request with it (a 502 whose detail is the wrapper's banner). The mitochondrial chromosome has the same shape of problem. `prepare` for the GEMMA engine now writes the bimbam pair without the Y and M markers (cached beside the individual subset, line-aligned so the geno and snps stay in step), and the note says what was left out: "46 Y/M markers left out (GEMMA's LOCO cannot fit them)". DOL maps: 106 515 rows in 49 s. **And the four tests that had been failing since before this work**, all stale expectations rather than product bugs: * `benchmark_questions` pinned two traits to "page 2" of a 100-per-page search when they rank ~190th and ~219th of 246 -- one re-index pushes them over. It now walks up to three pages and asserts presence and LOD, not the page; and the helper `search()` pages, because a benchmark row can name a trait that ranks outside the first hundred of its own query. * `no_db_mode_degrades_gracefully` asserted exactly one hit for "alcohol" from the index alone. The index grew; the behaviour under test is that the search still answers, within the limit, with the database-backed fields null. That is what it asserts now. * `search_proxy_scenarios` and `search_parity_replay` never pinned the engine they test, so an ambient `SEARCH_ENGINE=native` made the proxy scenarios answer from the local index: a 200 where a 422 was expected, and "expected 2 upstream calls, got 0" for every replayed query. Both now set `SEARCH_ENGINE=forward`. * src/genotypes.rs: `DEGENERATE_CHROMOSOMES`, `drop_degenerate_chromosomes`, `Prepared.markers_dropped`. * src/routes/gemma.rs: also the 500 that `eb5320d` introduced -- counting parents/F1s from `panel - keep` instead of subtracting the exclusion *set*, whose length can exceed the panel (subtract overflow). * scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: na.strings scoped to the genotype columns. * tests/{benchmark,no_db,search,search_parity}.rs: as above. Verified through the API on DOL trait 10003: R/qtl 112 913 rows, GEMMA 106 515 rows. gemma/rqtl/fitqtl suites green (14, 10, 4).
9 daysgemma: leave out individuals with no genotypes, which make the kinship singularPjotr
Asking whether the singular matrices could be fixed turned out to have two answers, one of them a real bug on our side. The bug: a panel can name individuals whose genotype row is empty, and an empty row makes the kinship matrix singular. GEMMA then reports "eigenvalues close to zero" and dies in LAPACK with `LUDecomp ... input domain error` -- a message about linear algebra for a problem that is a missing genotype. Measured on this deployment's own files: BXD-AE.4 136 of 344 individuals have no genotypes MDP 93 of 178 BXD-Heart-Metals.4 200 of 481 HET3-ITP.3 2867 of 2867 (the whole file is empty) Dropping them makes those panels map: BXD-AE.4's GWA step goes from a failure to 3.2 s, and MDP's kinship step from a failure to success. The individuals contribute nothing to a scan either way, so nothing is lost; `notes` counts them, and the "Genotype:" line no longer blames parents/F1s for them (`prepared.excluded` is "panel minus kept", which now includes both, so the two counts are separated). Not the cause, but worth naming: `AXB` fails earlier still, on its annotation -- `ERROR: Enforce failed for D1Mit296 in src/gemma_io.cpp ... ReadFile_anno` -- which is a broken `.snps` file, not a matrix. That wants a pre-flight check of the annotation (report the offending marker) rather than a GEMMA failure, and is not in this commit. Considered and rejected: a ridge on K (`K + epsilon*I`) would work mathematically but GEMMA has no option for it and the wrapper caches K by content hash, so it would mean patching the wrapper's cached matrix; `-gk 2` and dropping markers do not add rank; and the non-LOCO path is the one that does not run at all here. * src/genotypes.rs: `individuals_with_genotypes`, one pass over the bimbam file marking the individuals with any genotype at all. * src/routes/gemma.rs: filter `keep` by it before alignment (so the trait values drop with them), and say so in `notes`. * tests/gemma.rs: `individuals_with_no_genotypes_are_left_out` -- BXD-AE.4 maps, the note names the drop and the reason, and the genotype line no longer claims 344 individuals. * README.md: the behaviour and why it exists, under /compute/gemma. 307 tests pass; the same four as before do not.
9 daysrqtl: map the cM-only genotype filesPjotr
44 of this deployment's `.geno` files carry `cM` and no `Mb` column, and none of them could be scanned. GeneNetwork3's wrapper (copied verbatim here) assumes the position columns are `Chr, Locus, cM, Mb` and then starts the samples at a hardcoded column 5. On a cM-only file that is one column too far, so the genotype block loses its first individual and the whole scan dies in `do.call(rbind, ...)` with "number of columns of result is not a multiple of vector length" -- a message that says nothing about the cause. The wrapper now picks the map column from whichever of the requested scale and the file's own columns exists (falling back, and saying so), and starts the samples right after the position columns; the sex default and the sample-count guard use the same count. `scripts/qtl_fitqtl.R` had the same assumption and got the same fix. This is the second deliberate deviation from GeneNetwork3's script, after the sample-count guard, and both are commented as such at the code. * scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: the position column and the first sample column are computed, not assumed. * tests/test_data/rqtl_sample1_cm.geno: the existing fixture with `Mb` dropped -- same panel, cM only. * tests/rqtl.rs: `a_cm_only_genotype_file_maps` scans it and asserts all six markers come back (the failure mode was five-vs-six columns, not an error message that pointed at the genotype file). * src/rqtl.rs, src/routes/rqtl.rs: the docs said the script never forwards `--scale`; it does, and now falls back, so the request field says what `pos` is in. Verified through the API: DGRP (cM-only) now maps (3 markers), and AXB (499 markers) and BXD.4 (3 811) are unchanged. 307 tests pass; the same four as before do not.
9 dayscompute: fit covariates in /compute/fitqtl tooPjotr
`fitqtl` takes a `covar` argument, so the effect it reports can be the one *after* conditioning -- which is usually the number worth quoting for a peak that overlaps a known confounder. `covariates` now works on /compute/fitqtl exactly as on /compute/rqtl: the same `covariates::resolve` (case attribute by name, inline values, explicit map, same coding and same note), so the two endpoints report a covariate identically. The R side does what R/qtl needs: the phenotype file carries one column per covariate after the trait, `--covarstruct` says which are categorical, and the script dummy-expands those (k-1 against the first sorted level, missing values left missing). Two things it has to get right: * `model.matrix()` drops rows with a missing value, which silently misaligns the covariate frame against the cross -- so the dummies are built directly (`ifelse(is.na(x), NA, x == level)`), not through `model.matrix`; * a covariate can be perfectly spread across the panel and **constant among the individuals a trait covers** -- BXD's `Status` is exactly that -- and R/qtl then stops with `Lapack routine dgesv: system is exactly singular`. Columns that do not vary among the phenotyped individuals are left out, and the reply names them, the same rule the GEMMA covariate file applies. Measured on BXD trait 10001 at its peak, `BXD.4`: no covariate LOD 2.9310 additive +2.394 32.8% var with Status LOD 3.3602 additive +2.463 36.6% var with an inline numeric covariate LOD 2.9373 additive +2.361 The LOD is no longer the scan's once a covariate is fitted -- it is a different model -- which the docs now say, next to the no-covariate parity check that still holds. * scripts/qtl_fitqtl.R: `--covarstruct`, dummy expansion, the degenerate column rule, and the dropped names in the summary line. * src/fitqtl.rs: writes the covariates into the phenotype file and the struct file, and passes both. * src/routes/fitqtl.rs: the `covariates` field, resolved after alignment so the coding and the complete-case count match /compute/rqtl. * tests/fitqtl.rs: a covariate run (the note, the fitted row, a changed LOD) and an unknown attribute refused with the list of valid ones. * README.md, docs/www/SKILL.md: covariates on this endpoint, and why its LOD stops matching the scan's. 306 tests pass; the same four as before do not.
10 dayscompute: add /compute/fitqtl for effect sizes at named markersPjotr
`/compute/rqtl` answers "where" -- scanone gives a LOD per marker -- but not "how much": a scan statistic is not a fitted model. R/qtl's `fitqtl` fits one, and reports the QTL's effect, its standard error, the model LOD and the variance explained. That is what a table of effects needs, and it is a separate endpoint because it is a separate question, asked about a handful of markers (a scan's peaks) rather than a genome: `fitqtl` fits one model per marker, so a genome-wide version would be thousands of fits. Cap: 100 markers per request. Rows are positional: marker, chr, pos, lod, pct_var, additive, se, t, p `additive` is fitqtl's estimate for the QTL term -- R/qtl codes the QTL genotype +-1, so it is half the difference between the two homozygote classes, the same convention GeneNetwork's published `additive` uses. **Two independent checks that the numbers are right.** On BXD trait 10001 (`BXD.4`), at the scan's peak: * `lod` = 2.930985, and `/compute/rqtl`'s scanone gives 2.930985 at the same marker -- identical to every digit the CSV carries, because they are the same cross and the same phenotype. That parity is now an integration test; * `additive` = 2.394472, against the value genenetwork.org publishes for that trait (`/api/v_pre1/trait/BXD/10001` -> 2.394444), which is 28 ppm away -- and that published number came from qtlreaper, a different engine. * scripts/qtl_fitqtl.R (new): reads the same `.geno` and two-column phenotype file as `rqtl_wrapper.R` (which is a verbatim copy of GN3's script and so not extended), builds the cross the same way, fits `fitqtl` at each named marker with `get.ests=TRUE`, and writes one CSV row per marker -- plus a one-line summary naming the strongest fit. `calc.genoprob(step=0)`: marker-level probabilities are all a marker-level fit needs. * src/fitqtl.rs (new): runs it, parses the CSV into positional rows (`chr` stays text, it can be `X`), keeps the command line for `notes`. * src/routes/fitqtl.rs (new): the endpoint -- same alignment rules as `/compute/rqtl` (by name onto the panel, parents/F1s excluded unless asked), the same `normalize`/`winsor_*` pipeline (effects come out on that scale), and the same R/qtl gate, so a fitqtl request and a scan queue against one another rather than competing for cores. * src/config.rs: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`). * tests/fitqtl.rs: the fixture (columns, sign of the effect, notes), the LOD parity against scanone, and the refusals (no markers, both genotype inputs). * README.md, docs/www/SKILL.md: the endpoint, what each column means, and when to reach for it instead of the scan. 305 tests pass; the same four as before do not.
10 dayscovariates: refuse a mostly-numeric attribute, and teach the skill to use themPjotr
Fitting `SeqCvge` as a covariate came back as `502 gemma_failed` with a message about LU decomposition: its values are depths (`40`, `41`) with a few two-library entries (`36+39L`), so the attribute was coded as **twenty categories** and GEMMA died on the resulting dummies. The failure named GEMMA, which is the wrong place to look. `code_case_attribute` now refuses that shape at the covariate step, naming the values and what to send instead: 422 invalid_covariate: case attribute 'SeqCvge': the attribute mixes numbers with number-like text (14 of 19 values look numeric, but e.g. '36+30L' does not), so coding it as categories would fit a quantity as a factor -- and GEMMA fails on the result. Pass an explicit `map` giving each of those a number (there are 5 such value(s): ...), or send the numbers inline. The rule is deliberately narrow: an odd level with *no* digits (`1, 2, unknown`) is still a level and still coded, so nothing that worked stops working; only dirty numbers are caught. Also, the skill's covariate coverage -- which the review above found thin: * when a covariate is worth trying (metadata that could confound: sex, age, epoch, batch, sequencing depth), and the covariate step in the session outline; * read `.data.descriptions` before choosing one -- they are the only place that says `SeqCvge` is sequencing depth and `Epoch` a family subgroup -- and the endpoint table now says so; * read the covariate line in `notes` for the coding and the complete-case n, and report that n beside any LOD/p change; plus `Cyro`/`Cryo` being two levels in BXD; * the categorical-vs-numeric rule measured rather than asserted: `RRID` refused (82 distinct), `Status`/`Epoch` accepted, and the compound-value trap written up with the `map` workaround; * GEMMA's dummy columns, the column it drops when a level has no fitted individual, and that R/qtl's note has no such line; missing values dropping individuals; the 8-covariate cap and the `group` requirement. * src/covariates.rs: the refusal, with a unit test for both sides of it; * tests/gemma.rs: `SeqCvge` is a 422 naming `map`, not a GEMMA failure; * docs/www/SKILL.md: the covariate section, the session line, and the case-attributes rows in the endpoint table. 300 tests pass; the same four as before do not.
10 daysapi: normalize trait values, and winsorise the tails, before mappingPjotr
GeneNetwork2's trait page offers a set of normalisations, and its mapping page warns about outliers without ever fixing them. This adds both, under one spelling that `/compute/rqtl`, `/compute/gemma` and `/groups/{g}/traits/{id}/values` all share: normalize none | log2 | log10 | sqrt | negate | zscore | robust_zscore | rank (GN2's names `invert` and `qnorm` accepted) winsor_quantile clip each tail at a fraction, e.g. 0.05 winsor_fences clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule They run in a fixed, reported order -- transform, then winsorise, then standardise -- so clipping acts on the scale being analysed and the result lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The note gives the steps *in the order they ran* (`winsorized at Tukey fences, then zscore`), the count clipped in each tail, and how many values sit outside Tukey fences -- the warning GN2 gives before mapping and this API gave nowhere. **Checked against genenetwork.org itself.** GN2's public trait page embeds its own reference values (`js_data.qnorm_values` / `zscore_values`, the `data-qnorm`/`data-zscore` attributes), so there is an oracle for the two transforms whose definition matters. On BXD trait 10010 (28 samples): `rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's three-decimal rounding. Getting there found a real disagreement: GN2 uses `scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use the **population** sd; ours used the sample sd and came out `sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2. Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based inverse-normal (Blom) transform, so it is called `rank` here; and its `sqrt` silently adds 1 when a value is in [0,1], which shifts every effect size -- a negative value is refused instead. * src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom scores over an Acklam inverse-normal, `Params` for query/body use, `to_json` for suggestions, and eleven unit tests -- the quantile table, ties sharing an average rank, missing values never imputed, GN2's own `find_outliers` example, and the note's wording. * src/routes/{rqtl,gemma}.rs: the three fields, applied **after** alignment (over the samples that enter the fit) and before anything reads the values, with the note; the R/qtl permutation suggestion and GEMMA's R/qtl suggestion carry the pipeline. * src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values endpoint, and the mapping bodies in `actions` carry the same fields -- a caller who fetches z-scores must map those numbers, not the raw ones. * tests/groups.rs: values are mean 0 / population sd 1, the note says so, every `/compute/*` suggestion in `actions` carries it, and an impossible pipeline is a 422 `invalid_normalization`. * README.md, docs/www/SKILL.md: what each name means, the order, the population-sd parity, and that effects come out on the transformed scale. 298 tests pass; the same four as before do not.
10 dayscompute: report GEMMA's effect size next to the scorePjotr
A GEMMA row now carries the effect beside the p-value it belongs to: `… beta, se, l_mle, additive, p_lrt`. `additive` is `-beta/2`, which is what GeneNetwork's own tables and CSV call the additive effect (gn2/wqflask/marker_regression/gemma_mapping.py:240). Halved because half the difference between the two homozygotes is the usual additive effect; negated because GN2's BIMBAM conversion codes `@mat` as 1, so GEMMA's beta is per *maternal* copy while GeneNetwork's published sign is anchored on the other allele. Two facts that make this worth doing carefully: * `beta` and `se` are left exactly where GEMMA put them. `se` is what makes the effect interpretable (`beta +/- 1.96*se` is a 95% interval) and both are GEMMA's own numbers, so replacing them with a derived one would lose information; * the derived column goes *before* the p-value, not after it. `p_lrt` must stay the last column: `gemma::peak_marker` reads `row.last()` for the peak (and with it the PheWAS and plot suggestions), and the plotters and the docs all say "the last column is the p-value". Appending would have silently pointed every peak at an effect size. The response also says what the three columns mean, and that they describe *this run* on these samples and this genotype version -- not the `additive` trait info reports, which came from the published mapping (trait 20010: this run -0.54, published +2.118, same peak marker). Nothing is added when GEMMA wrote no `beta` (a non-LMM run): the columns are then exactly GEMMA's. * src/gemma.rs: `with_effect_size`, applied by `parse_assoc_file`, so /compute/gemma and /compute/gemma/ravanan both get it. * src/routes/gemma.rs: the note. * tests: four unit tests (placement, sign and halving, `peak_marker` still finds the peak from the last column, a table without beta untouched, a null effect for a row with `NA` beta) and one end-to-end test asserting the column sits immediately before `p_lrt`, every value is `-beta/2`, `p_lrt` stays a probability, and the note is there. * README.md (the response example and what the column means), docs/www/SKILL.md (how to read and quote it). 286 tests pass; the same four as before do not.
10 dayscompute: let GEMMA name the trait, so its plot can be titledPjotr
Two GEMMA runs over one BXD panel drew genuinely different pictures -- traces differ on all 20 chromosomes, peaks -log10 p 5.48 at chr8 97.0 Mb against 5.43 at chr1 175.6 Mb -- and both came out headed **"trait gemma: GEMMA association along the genome"**, because GEMMA's request had no trait label at all: `/compute/gemma` fell back to `<group>::gemma`, and `/plot/gemma` fell back to "gemma". The plot of each trait was told apart by its annotation and nothing else. `trait_name` is now an optional field on `GemmaRequest`, used for the `Trait: ...` note (first in `notes`, as on `/compute/rqtl`), for the PheWAS and R/qtl suggestions, and for the `/plot/gemma?trait=` suggestion -- so a posted response titles its own plot, and `?trait=` still overrides. Without it nothing changes: the label stays `<group>::gemma`. * src/routes/gemma.rs: the field, the label resolution, the note, the plot href. * tests/gemma.rs: the covariate run names its trait and asserts the note is first; tests/plot.rs: a GEMMA plot titles itself from the note, and an explicit `?trait=` wins over it. * README.md (`/compute/gemma`, and the plot section), docs/www/SKILL.md (the GEMMA options list, and what the plot title comes from). 281 tests pass; the same four as before do not.
10 daysapi: name the caller on the response, so the log can say whoPjotr
Answering "is robw using this successfully, and the others?" from the access logs took an hour of inference -- user agents, search terms, which traits were mapped -- because nginx records the address and the subject is only in the token. Two callers can share an address (an agent and a person behind one NAT), one caller can move between addresses (an LLM's egress is a /24), and the answer for a given person was "probably 80.115.1.132". The API now sets `x-gn4-user: <token sub>` on every **authenticated** response, and the `gn4` log format records it as `user=`. The subject is only known inside the handler (the `AuthToken` extractor verifies the token there), so a middleware opens a slot in the request extensions, the extractor fills it, and the header is set on the way out; a request with no valid token gets no header, which is the honest answer and reads as `user=-`. The caller only ever sees an identity it already presented. * src/auth.rs: `USER_HEADER`, `CallerSlot`, and the extractor recording its subject. * src/lib.rs: the `tag_caller` middleware, outermost so the slot exists before any extractor runs. * tests/auth.rs: `an_authenticated_response_names_its_caller` -- the header equals the token's `sub`, and `/version` (no identity) carries none. * etc/nginx-vesuvio.pu.ac.ke.conf (and the deployed copy of it): `user=` in the log format. * README.md, docs/nginx.md (the format, an `awk` one-liner for per-user counts, and the caveat that old and new lines differ in field count), docs/www/operator-checklist.md (`user=-` means no token, not a token that failed late). 281 tests pass; the same four as before do not.
10 daysplot: say which trait a plot is, not just draw itPjotr
Two traits over the same panel really do draw two different pictures -- verified on BXD.4, where all 20 chromosomes' polylines differ and the peaks are LOD 2.93 at chr8 69.4 Mb (mCV24409492) against LOD 4.78 at chr8 95.7 Mb (rs3666069) -- but you could not tell from the title: every plot came out headed **"trait rqtl: LOD along the genome"**, because `/plot/rqtl` labelled it from `?trait=` and nothing else, defaulting to "rqtl". The only visible difference was the peak annotation, and a caller posting the saved response back (which is the documented way) could not know the label was missing. The response now names what it mapped: `/compute/rqtl` writes `Trait: <name>` as its **first** note, and `/plot/rqtl` and `/plot/gemma` take the title's label from that note when `?trait=` was not given. The `POST /plot/rqtl` suggestion in `actions` also carries `?trait=`, so a caller that posts only `data` still gets the right title. * src/routes/rqtl.rs: the `Trait:` note, ahead of the column note; the plot suggestion's href names the trait. * src/routes/plot.rs: `trait_from_document`, used by both plot endpoints. * tests/plot.rs: `two_traits_draw_two_different_plots` -- two canned responses differing only in LOD, asserting each title names its own trait from its own note, each annotates its own peak and not the other's, and every polyline differs. That is the regression this is about: a plot that reused anything between requests would draw the first trait's shape for the second. * tests/{rqtl,phewas}.rs: the plot suggestion is now matched by prefix, with the trait asserted. * README.md, docs/www/ai-instruction.md: where the title comes from, and to post the whole response (or add `?trait=`) if you post only `data`.
10 dayserrors: log every 5xx, and remember the last database failurePjotr
A user reported that the token was fine and the service was up, but "everything backed by the relational database is failing" -- while `/search` (Xapian) and `/version` answered. By the time I looked, every data endpoint answered 200 and `/version` said `database: reachable`, and there was **no record anywhere** of what they had hit: the service logged nothing at all, so a caller's report could not be matched to a cause, and `/version`'s `database` is a *now* probe that says nothing about the minute before. Two things now make that diagnosable: * every 5xx is written to stderr as one JSON line -- status, code, operation_id, **request_id**, took_ms and a truncated detail -- so "500 at 12:31, request id ..." can be looked up. 4xx stay unlogged: they are the caller's business, and a 401/422 stream would drown this; * the last database failure is remembered (`db::record_failure`, called from `ApiError::from_db`, the one place every database error passes through) and `/version` reports it in `notes` for ten minutes, with its kind (`unavailable`, `connection_lost`, `query`) and the driver's message. That is the gap that misled the user: a healthy probe beside failing endpoints looks like an auth problem from outside. `is_connection_loss` gained a `_detail` form so the caller that already has the rendered message does not render it twice. * src/error.rs: the log line, and recording failures by kind; * src/db.rs: `FailureKind`, `Failure`, `record_failure`, `recent_failure`, and two unit tests (a failure is remembered and ages out; a driver message is classified correctly); * src/lib.rs: the `/version` note, absent unless something failed recently. Checked against a deliberately broken `SQL_URI` on a scratch instance: the data endpoint answers `503 db_unavailable`, the log carries the request id, and `/version` shows the note while still answering 200.
10 dayscompute: gate GEMMA from 50 individuals, R/qtl from 1 000Pjotr
One threshold for both engines was the wrong shape. For R/qtl the interesting line is where a `scanone` stops being seconds (a few hundred individuals is seconds of one core), so 1 000 stays. For GEMMA it is much earlier, and not because of the runtime: - every GEMMA run goes through **one shared cache directory** (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome kinship archives of hundreds of megabytes by content hash. Two runs at once are two writers over the same archives, and a half-extracted kinship another run reads surfaces as a GEMMA failure; - a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382 individuals write 207 MB of kinship per chromosome. So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every mapping request is gated in practice, and only a handful of individuals (a four-way cross's parents, a probe of the pipeline) runs free. `ravanan` uses the same number, since it writes into the same cache. The threshold moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint carries its own; the policy, the slots and the 409 are unchanged. Tests, asked and answered: at 50 the three gemma tests that do real work (fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated and share one token, so on a cold cache two of them at once would collide into `409 compute_busy` -- a gate doing its job, failing a test that only wanted to map something. They now take a lock, and only they do; the validation tests stay parallel, and the rqtl suite is untouched (its gate still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an empty directory, `cargo test --test gemma` is green twice in a row, 18 s per run, i.e. really recomputing. * src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL` (50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`; the test that pinned one number now pins both and that each endpoint is wired to its own. * tests/gemma.rs: the three real-job tests take the lock, and say why. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: the two thresholds and the cache-directory reason for the lower one. 277 tests pass; the same four as before do not.
10 daysapi: describe case attributes, and gate large computations onlyPjotr
Two things that came out of reading a real response rather than a test. **Say what a case attribute means.** `GET /groups/{group}/case-attributes` returned names and values, and the per-trait attribute endpoint returned values, so a caller had to guess that `SeqCvge` is a sequencing depth. MariaDB has had the answer all along in `CaseAttribute.Description`: SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016 Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct Epoch BXD family subgroups. Each number with common parents. ... Sex F= female, M= male Both endpoints now carry it -- a `descriptions` map on the group listing, a `description` field on the per-trait one -- and the covariate notes quote it, so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for 1 male of each strain in 2016): categorical ..." instead of leaving the name to be interpreted. It matters most for covariates: `SeqCvge` is a property of the *genotype* data, its values include compound ones (`40+31L`, two sequencing runs), and the server codes those as 20-odd levels rather than as a depth -- a covariate fitted for the wrong reason, which the description is the first hint against. **Gate large computations, not every computation.** The gate was applied to every job, so a 93-strain test request could be refused because somebody else's 6 000-individual run held the slot -- which is how the test suite started serialising itself behind a mutex, a workaround for the wrong problem. What the gate is for is *large* work: a BXD panel of ~200 finishes in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000 individuals): below that no slot is taken, nothing waits and nothing is refused; above it the old policy holds unchanged (1 slot for gemma, 1 for ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409 naming the run in the way). The size is not known when the request arrives -- the panel is resolved server-side -- so the decorator now makes a *light* claim and the handler promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the point where it knows, then hands it to `Gate::run` around the call that computes, so the slot still outlives a client disconnect. A light claim registers nothing and its drop removes nothing: only a held claim may touch the claimant map. * src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`, `Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved out of `claim_or_busy`), and `Gate::run` unchanged in spirit. * gnauth-macros: binds the light claim; the body keeps one gate call. * src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is known, and the compute call wrapped in `run`; error mapping moved inside the job so the future returns `ApiError`. * src/rqtl.rs: `Sample` is `Clone` (the job owns its copy). * tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are under the threshold); rqtl keeps its lock only for the machine's sake -- concurrent `Rscript` scans would compete for cores and scratch, not for a slot. Four new gate tests pin the threshold, that a small job never takes the slot, that one caller cannot hold two large jobs, and that another caller waits rather than fails. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: what the threshold is and why it is 1 000. Measured on :3001 after the change: two concurrent small-panel GEMMA requests from one token both answer 200 (one was 409 before), while two 6 382-individual requests answer 502/409 -- the second gated with the usual `compute_busy` naming the first. 277 tests pass; the same four as before do not.
10 dayscompute: say what each covariate note is actually aboutPjotr
The shared covariate note promised "5 dummy column(s) named Status.<level>" on both compute endpoints, but only one of them gets dummies: GEMMA is handed k-1 columns and R/qtl's wrapper is handed the single level column and expands it itself. On `/compute/rqtl` the sentence described something that never appears -- caught by reading a live response on production, where the note said "5 dummy column(s) named Status." one line above the Rscript command line that passes no such thing. The note is now engine-neutral -- `... 90 of 93 samples have a value, 34 with both the trait and the covariate; 6 levels` -- and GEMMA says what it was handed in its own note, which is the part a reader comparing two runs wants: `GEMMA fits 4 covariate column(s): Status.Live, Status.UTHSC, Status.UTHSC2, ...`. The unit test asserts the level count, and the end-to-end test asserts the columns note names `Status.` columns twice or more, so a covariate silently fitted as one ordinal number fails it. * src/covariates.rs (resolve), src/gemma.rs (compute), tests/gemma.rs, docs/www/ai-instruction.md (the example note).
10 dayscompute: refuse GEMMA permutations instead of ignoring themPjotr
`gemma-wrapper --permutate n` repeats the *whole* LOCO association scan per permutation -- it shuffles the phenotype file and starts over -- so a thousand permutations is thousands of full scans: hours to days on a real panel, against the single slot `/compute/gemma` has. That is not a test this deployment carries, and an assistant asked for significance thresholds will ask for it, so the answer has to be an explicit one. Until now the request type simply had no such field and serde ignores unknown keys, which is the worst of the three outcomes: the caller gets a 200, no thresholds, and no indication that what it asked for was dropped. `nperm`, `permutations` and `permutate` are now named fields and refused with `422 permutations_not_supported`, naming the cost, pointing at `POST /compute/rqtl` with `nperm` (the endpoint that does have the test, over the same panel and values) and at the `actions` member every successful GEMMA response already carries, pre-filled with exactly that call. Zero keeps its usual meaning -- off -- so a client sharing one body shape across the two endpoints still works. * src/routes/gemma.rs: the three fields, the refusal before anything expensive runs, and the response note now saying permutations are refused rather than merely absent. * tests/gemma.rs: `compute_gemma_refuses_permutations` pins all three spellings, the code, and that `nperm: 0` is not a request for any. It takes the compute lock: the gate answers before the handler does, so a concurrent job in the same test binary came back 409 instead. * README.md, docs/www/ai-instruction.md: what to do instead -- judge a peak against the smallest `p_lrt`, and get thresholds from R/qtl. 273 tests pass; the same four as before do not.
11 dayscompute: fit covariates in GEMMA, and share them with R/qtlPjotr
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl` does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample numbers -- and fits it with `-c` on the association step only. That needed a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample names, positional over the genotype panel, numbers only. GEMMA has no factors, so a categorical covariate is dummy-expanded here into k-1 columns named after their levels (`Status.Live`, the first level being the reference), where R/qtl's wrapper does that expansion for itself. The spec shape, the alignment onto the panel and the text coding are now one module, `src/covariates.rs`, called by both routes; `/compute/rqtl` behaves as before except that its note also reports the complete cases (`... 34 with both the trait and the covariate`), which README and the instructions had been promising all along. A missing covariate value is written `NA`, which GEMMA treats exactly as it treats a missing phenotype -- that individual leaves the fit (measured: one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's `-9`, which GEMMA fits as the value minus nine. Found on live data, not in the fixtures: a level that exists in the panel but that none of the *fitted* individuals carries becomes an all-zero dummy column, and GEMMA stops with `LUDecomp ... input domain error` rather than dropping the term the way R would. Columns that do not vary among the individuals with both a trait and a covariate are now left out, and the response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a covariate left with no columns runs without one, with a note saying so. Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes its file arguments, so a different covariate set is a different cache entry while the kinship already computed is still reused. Covariates need a `group`: individuals are aligned by name, which an explicit `genotype_file`/`snps_file` request does not carry (`422 covariate_unavailable`), and an attribute the group does not define lists the ones it has. * src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now `CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`, `Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file` (moved out of src/rqtl.rs) and `write_gemma_covariates_file`. * src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate file, `-c` after `-a` and before `-lmm`, and the notes about dropped individuals and dropped columns. * src/routes/gemma.rs: the `covariates` field, resolved after the trait values so the note can count complete cases; `RqtlCovariate` replaced by the shared spec. * tests: six new -- the GEMMA file (numerical with `NA`; two and three levels; inline codes; a dropped degenerate column) and an end-to-end `compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with `Status`, which also pins that `-c` is in the GWA command and not in the kinship one, and that the covariate changes the fit rather than reusing the plain run's cache. The gemma tests now take a lock: the gate answers a second job from the same caller with 409 instead of queueing it. * README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl points at the shared shape and explains `x` vs `NA`), docs/www/ ai-instruction.md (both endpoints; what differs in `notes`). Also in the instructions: `/version` reporting a `git` value you have not seen before now says explicitly to pull the repository and re-read `docs/www/ai-instruction.md` before the next call. This deployment is updated in place and a stale page does not announce itself -- the API accepts the request without the new option, or answers with fields the old text never mentions. 272 tests pass; the four that do not (benchmark, no_db, and two search parity/proxy replays) fail identically at the parent commit.
11 dayscompute: gate ravanan at one job, and name the binary that failedPjotr
`POST /compute/gemma/ravanan` is now gated like the other two: one workflow at a time, one per caller, `409 compute_busy` naming the run and linking `/compute/gemma/status/{id}` (ravanan has no status route of its own). One slot because ravanan already fans its per-chromosome steps out -- over Slurm, when that backend is selected -- so a second request would only queue behind the same scratch directory. Its gate is separate from `/compute/gemma`'s: different runners, different cache directories, so one of each at once is the overlap `/compute/rqtl` is already allowed. Verified with a stub `CCWL_CMD` that sleeps: the holder's request held the slot for the full 5 s, the same caller's second request was refused at once, and another caller's waited for the slot and then ran. While testing that, `GemmaError::Io` turned out to print "failed to run gemma-wrapper" for *any* I/O failure -- a missing `CCWL_CMD` produced "failed to run gemma-wrapper: No such file or directory", which sent me looking at the wrong binary. Spawn failures are now their own variant naming the command, and plain file I/O says "I/O error: …". * src/compute_gate.rs (ravanan, permits): `Gate::new(1)`; `permits()` exposes the slot count, and a test pins all three endpoints' counts (gemma 1, ravanan 1, rqtl RQTL_MAX_CONCURRENT). * src/routes/gemma.rs: the decorator on compute_gemma_ravanan, with `status = "/compute/gemma/status"`. * src/gemma.rs (Spawn): names the command; the four run_streaming call sites use it; unit test for both messages. * README.md (/compute/gemma/ravanan), docs/www/ai-instruction.md (the slot list is now 1 / 1 / 8), docs/www/operator-checklist.md: concurrency.
11 dayscompute: gate R/qtl at eight concurrent scans, one per callerPjotr
`POST /compute/rqtl` now takes up to `RQTL_MAX_CONCURRENT` slots (default 8, read when the gate is built) with the same one-job-per-caller rule as GEMMA: a second submission from the same token while its scan is running or queued is refused `409 compute_busy`; a submission that finds every slot taken waits for one to come free. Eight because an R/qtl scan is a single-threaded Rscript on a 96-core box; not more, because a scan on a 3 000-individual cross is minutes of CPU and the same machine serves the database. Unlike GEMMA's single slot, concurrent scans share no cache (R/qtl results are uncached), so the only cost is the CPU they take. Two flaws only appeared once there was more than one slot: - the "what is running" record was a single `Option<Active>`, so a refusal quoted whichever job had most recently taken a slot -- robw's 409 named JoM's run id. It is now a map keyed by subject, and `a_refusal_quotes_the_callers_own_run_id` pins it; - `Busy::describe` took the endpoint identifier, so the message read "your rqtl job"; the decorator takes a separate `label` ("R/qtl") from the status path. The test suite hit the rule itself: six compute tests in tests/rqtl.rs share one token, and cargo runs them on several threads, so three were refused instead of run. They now serialise on a module mutex, with a comment saying why -- the same lesson clients need, that parallelism means more tokens, not more requests from one. * src/compute_gate.rs (rqtl()): eight slots from `RQTL_MAX_CONCURRENT`; `Gate::new(permits)` was already parameterised. `running` becomes a map keyed by subject, since one record cannot name N jobs. Module doc talks about slots rather than a single slot. Tests: eight subjects run and the ninth waits, a caller still may not take two even with slots free, and a refusal quotes the caller's own run id. * src/config.rs: RQTL_MAX_CONCURRENT, default 8, fixed at first use. * gnauth-macros/src/lib.rs: `label = "…"`, so prose and status path are independent. * src/routes/rqtl.rs: the decorator, and `_token` becomes `token`, which it needs. * tests/rqtl.rs: compute tests serialised. * README.md (/compute/rqtl), docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: the numbers per endpoint, and that more parallelism means more tokens.
11 dayscompute_gate: undo the claim if the wait for the slot is cancelledPjotr
`claim()` inserts the subject into `claimants` and *then* awaits the permit, and that await is cancellable: a client disconnect while queued dropped the handler's future with the entry still in the set, so that subject was refused `compute_busy` for the life of the process. The reservation guard removes the entry when the wait is cancelled, and hands ownership to the `Claim` once the permit arrives -- the slot itself was never leaked (no permit had been taken), only the subject's name. Found by asking what could race; the new test fails without the guard (verified by disabling it) and passes with it. * src/compute_gate.rs (Reservation): the entry's owner from insertion to permit, `hand_over()` at the hand-off. (Drop for Claim): field order is load-bearing -- the maps are cleaned before `_permit` is released, so the next waiter cannot see a stale entry or a stale `running` record; and take the locks poisoned-tolerant (`lock_recover`) rather than silently skipping the cleanup. (tests): `a_cancelled_waiter_does_not_wedge_its_subject`.
11 dayscompute: one gemma job at a time, per caller, via #[compute_gate]Pjotr
`POST /compute/gemma` now takes a single slot: one job runs at a time across all callers, a second submission from the same token is refused immediately with `409 compute_busy` (whatever it would have waited for, its own job is already in the way), and a submission from another token waits for the running job and then runs. Nothing else is gated -- ravanan and /compute/rqtl are untouched. The gate is one `tokio::sync::Semaphore(1)` (fair: waiters take the permit in arrival order) plus a `Mutex<HashSet<String>>` of subjects holding the slot or queued for it, in a per-process static. The claim is moved into the spawned job task, so a client disconnect -- which does not kill the child process -- cannot hand the slot to somebody else while the CPU is still busy. The endpoint carries none of that: `#[compute_gate(gemma, run_id = params.id.as_deref())]` expands to a claim and a run, and requires the handler to take `Ctx` and `AuthToken` (every compute endpoint is token-only, and the token's sub is the key), so an endpoint that is not token-gated cannot be decorated. The policy -- the 409's status, code, wording and next link -- stays in plain Rust (`Gate::claim_or_busy`), so the gate's own tests cover it; the attribute is decoration over two calls. * src/compute_gate.rs: new. Gate, Claim (released on drop, including a panic), Busy, claim/claim_or_busy/run, and four tests, two of them on a four-thread runtime (the earlier ones ran single-threaded and so never exercised the cross-thread hand-off; one now asserts at most one task inside the slot across eight tasks). * gnauth-macros/src/lib.rs: the `compute_gate` attribute, alongside the `gnauth` markers it shares `syn`/`quote` with; arguments documented (gate, endpoint, status, run_id) for the endpoints still to be gated. * src/routes/gemma.rs: the decorator replaces 57 lines of claim and spawn plumbing in compute_gemma. * docs/www/ai-instruction.md (4a, 5): one job at a time per caller, and that a `compute_busy` 409 means poll, not resubmit.
11 dayshttp: accept bodies up to 64 MiB, not axum's 2 MiB defaultPjotr
A caller posting a mapping result back to /plot/rqtl -- or a compute request carrying a value per individual plus covariates -- was refused with `413 Failed to buffer the request body: length limit exceeded`. That is axum's DefaultBodyLimit, 2 MiB, which nothing here had raised: a real GEMMA run on the BXD WGS panel is ~1.9 MB, so the plot endpoints sat just under the cap and anything bigger (a genome-sized run, or every animal's covariate value) fell over it. nginx's 10m was never the binding limit. * src/lib.rs (BODY_LIMIT_BYTES): 64 MiB, with the reasoning and the note that a proxy has to allow as much or it refuses first. * etc/nginx-vesuvio.pu.ac.ke.conf, docs/nginx.md: client_max_body_size 64m, so the two limits agree instead of the proxy cutting in at 10m. * README.md ("Request body size"): both caps, and that a request does not need to be large -- `{"case_attribute": "Sex"}` has the server read the attribute and align it, where posting a value per animal is pure payload. Verified: the 11.9 MB body that was refused now returns an SVG; 67 MB still gets axum's 413.
11 daysdb: recover from a connection the server closed, and report the databasePjotr
Every database-backed endpoint 500'd with `CodecError { IO error: Broken pipe }` while the process answered /version quite happily. The cause was ours: `pool()` turns off the driver's `check_health` (a ping at checkout, dead connections replaced) to save a round trip -- sound against a database 130 ms away, worth nothing against loopback, and it costs the pool its self-healing. One server-side close of an idle session then becomes one failed request per dead socket: ten in a row, until a restart. * src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping; `is_connection_loss` recognises the errors that mean the peer went away (broken pipe, reset, went away, lost connection, closed); `forget_pool` drops the cached pool so the next call rebuilds it; `health` probes reachability, cached ten seconds. One pool static shared by pool()/forget_pool (they had drifted into two). * src/error.rs (ApiError::from_db): on a connection loss, forget the pool -- the one place every database error passes through, so recovery does not depend on each caller. * src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a service that answers while nothing behind it works is now visible. * src/config.rs: the switch, with the trade written down. * tests/db_recovery.rs: kills every pooled session from a second connection, then asserts the next twelve requests all succeed -- the scenario that produced the outage, now a regression test. * tests/...: unit test for the classifier (its own IO error text and the "this is a bad query, do not retry" cases). * README.md ("Database connections"), docs/www/operator-checklist.md: the knob, why it defaults on, and what /version reports.
11 dayscompute/rqtl: fit covariates, by case-attribute name or as numbersPjotr
R/qtl has always been able to fit covariates; the API now passes them through. `covariates` takes one or more specs, each either a case attribute of the group by name -- which the server reads, aligns onto the panel and codes -- or explicit per-sample numbers: {"case_attribute": "Sex"} {"case_attribute": "Diet", "map": {"HFD": 0, "LFD": 1}} {"name": "age_days", "kind": "numerical", "values": [...]} The coding is the server's job because the phenotype file is numeric: text with 2-20 levels is coded 0..k-1 in sorted order and declared `categorical` (the wrapper dummy-expands more than two levels), numeric text goes through as `numerical`, and one level, more than 20 levels, or an explicit map that misses a value is a 422 asking for a map rather than an invented ordering. An attribute the group does not define is a 422 that lists the ones it has. Missing values become `x`/NA and R/qtl drops those individuals. Everything that decides the model goes into the cache key -- covariate names *and* kinds, since a categorical covariate's dummy expansion is not the same model as a numeric one -- and the response says what happened: Covariate Sex: categorical coded F=0, M=1, 3281 of 3281 samples have a value, 3281 with both the trait and the covariate Measured on BXD-Longevity AgeAtDeath (3281 animals, WGS panel, 21056 markers): LOD 23.100 without the covariate, 23.522 with Sex, same peak (rs30847416, chr 1, 5.0 Mb), ~15s more for the covariate run. * src/rqtl.rs (Covariate, CovariateKind, code_case_attribute, write_covarstruct_file): the coding rules, the extra pheno columns, the covarstruct file, `--addcovar --covarstruct`, and the model's cache key; five unit tests. * src/routes/rqtl.rs (RqtlCovariate): the request shape, resolution through the group's case attributes, duplicate/limit/ambiguity checks (8 covariates at most), and the per-covariate note. * tests/rqtl.rs: a real covariate run on UM-HET3 (894 markers, so the test stays a test) asserting the coding note and the `--addcovar` line, plus the unknown-attribute 422. * README.md, docs/www/ai-instruction.md (4a): how to pass them, and that the note's coding and complete-case count are the things to read.
11 dayscase-attributes: one attribute for one trait, in that trait's sample orderPjotr
The trait-values endpoint stays about trait values; a covariate needs the study's metadata for the same samples, aligned the same way. So: GET /groups/{group}/traits/{trait_id}/case-attributes/{attribute} returns one attribute's values for exactly that trait's samples, **in the trait's own order**, which is what makes it usable without the caller re-sorting anything: /groups/BXD-Longevity/traits/10001/case-attributes/Sex -> 3281 values in AgeAtDeath's order, F x 3101, M x 180 -> order identical to /traits/10001/values (asserted in the tests) Alongside the values it reports `distinct` (each value with its count, commonest first) and `num_values`. That is the check that decides whether an attribute can serve as a covariate at all -- a constant one cannot -- and it saves a second request just to find out. Attributes are per group, so an unknown one is a 404 whose `next` is the group's attribute list: BXD has no Sex, BXD-Longevity does. * src/db.rs (get_trait_case_attribute, get_group_trait_samples): the group's attribute values aligned onto the trait's sample list, joining CaseAttribute/CaseAttributeXRefNew on (InbredSetId, CaseAttributeId) as the group endpoint does; empty text is reported as no value. * src/routes/groups.rs: the handler, token-gated, with the response shapes and the two `next` links (the trait's values, the group's attributes). * tests/case_attributes.rs: alignment against an independent /values response, `distinct` accounting, ordering, and the 404s for BXD (no Sex) and for a non-existent attribute. * README.md, docs/www/ai-instruction.md (3. Case attributes): the endpoint, and "ask per trait rather than sorting the group's attributes yourself".
11 daysplot: take a posted result, and never re-run the mappingPjotr
The plot endpoints ran the mapping themselves, which meant the picture could come from a different run than the one the caller was looking at, and cost a scan to draw. They now take the mapping's own response -- the JSON /compute/{rqtl,gemma} returned, posted back -- and hand it straight to the plotter, which is what the scripts were written for (`python3 plot-rqtl.py result.json -o out.svg --trait …`). Validation replaces the run: `/plot/rqtl` needs `data.markers` with positional 4+ column rows, `/plot/gemma` needs `data` rows *and* `meta.columns` (its plotter reads the column names to find `p_lrt`), and anything else is a 422 saying "post that response back". `/plot/rqtl` also accepts just the `data` member; GEMMA cannot, and says why. The mapping's `actions` suggestion now describes posting its own response and carries no body -- duplicating a 21 000-row response into a suggestion would double the envelope for nothing, and the caller already holds it. * src/routes/plot.rs: posted result, shape validation, `?trait`/`?title` and `?bonferroni=false` passthrough to the scripts; no compute imports. * src/routes/{rqtl,gemma}.rs: the plot suggestion loses its body. * tests/plot.rs: canned results instead of real mappings -- the tests now need only the plotter, no Rscript or gemma; covers both engines, the bare-`data` form, and the 422 for a body that is a request. * README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: "post the response back", and why GEMMA needs `meta.columns`.
11 daysplot: draw a mapping, and suggest it from the mappingPjotr
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a saved mapping response into an SVG Manhattan panel. They are now endpoints, and the mapping that produces the response suggests them: POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml POST /plot/gemma same body as /compute/gemma -> image/svg+xml Taking the mapping's own body rather than the mapping's output means one call gets you the picture, and the plot cannot describe a run you did not do: the handlers call the compute handlers and hand the resulting envelope to the script, exactly as the scripts' docstrings describe (they read `data.markers`, `data.permutation` and the notes' strain counts). The mapping is re-run, but through the engines' own caches -- seconds on BXD. The reply is an SVG, not the JSON envelope: an image is not JSON, and the plotter's one-line peak summary rides back in `x-plot-summary` so a caller that cannot see the picture still learns where the peak is. Both compute responses gain the matching suggestion: `actions` carries "the same request, plotted" (built from the request that ran, like the permutation suggestion), so an assistant can map, look, and only then decide whether the peak is worth the 100-permutation run. * src/plot.rs: run the configured script over the envelope, read the SVG back, clean up; a missing script is a 502 naming interpreter and script, not a panic. * src/routes/plot.rs: the two handlers (token-gated), delegating to compute_rqtl/compute_gemma so validation and alignment stay in one place; `x-plot-summary` sanitised into a header. * src/routes/{rqtl,gemma}.rs: the plot suggestion, before the permutation one. * src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER. * tests/plot.rs: the suggestion resolves to a complete SVG with a peak summary, and an invalid mapping request is the mapping's own 422. * tests/{rqtl,phewas}.rs: find suggestions by href rather than position. * README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md (new "Plotting": python3 must be on the server's PATH).
11 dayscompute: suggest 100 permutations after a scan, and say what GEMMA can doPjotr
A scan gives LOD scores; the thresholds that make them readable come from permuting the phenotype, which R/qtl supports (nperm) and this API already ran -- but nothing told a caller to run it. So an unpermuted R/qtl scan now ends with the same scan at nperm=100 in `actions` (a POST, so not a `next` link), rebuilt from what actually ran: group + version + the caller's own named samples, or the genotype file. The description says what the thresholds are for and that it costs about 100 scans. GEMMA is the honest half: it has no permutation test, and this endpoint passes it nothing that would create one, so there is no nperm to suggest. Its response says so in a note -- judge a peak against the smallest p_lrt, not a genome-wide threshold -- and its `actions` suggests the R/qtl permutation run over the same group and values instead. That is real and runnable rather than a flag that would do nothing. (gemma-wrapper does have `--permutate n`, which shuffles the phenotype file and writes a permutation-stats file; wiring it up is a separate job -- docs/genotypes.md and the README say it is not passed.) * src/routes/rqtl.rs: the nperm=100 suggestion when nperm was 0, built from the request that ran (original named samples, not the aligned panel-length vector -- thousands of entries on a real panel). * src/routes/gemma.rs: the note, and the cross-engine suggestion. * tests/rqtl.rs: a path-based scan's suggestion reuses the same files. * tests/phewas.rs: a group-scoped scan suggests both -- the PheWAS in `next` (resolved) and the permutation run in `actions`. * README.md, docs/www/ai-instruction.md (4a): what thresholds are, that R/qtl offers them and GEMMA does not, and what to do about it.
11 dayscompute: suggest a PheWAS at the peak, and the plot that followsPjotr
A mapping's job is not finished when the rows come back: the next two questions are "what else peaks here?" and "show me". So a group-scoped mapping response now ends with both -- - `next`: `/groups/{group}/phewas?locus=<peak>&window=10000`, the peak being the strongest LOD (R/qtl, column 4) or the smallest p_lrt (GEMMA, last column). A GET that resolves 200, which is what `next` promises; measured: R/qtl's peak rs48756159 -> 3 traits, GEMMA's rsm10000005700 -> 12. - a note saying which two columns to plot: chr against LOD (R/qtl) or against -log10 of p_lrt (GEMMA). Rows are positional with meta.columns naming them, so plotting needs no reshaping -- the API does not render images, and saying exactly what to plot is more useful than pretending to. Path-based requests get neither: without a `group` there is no group to PheWAS over, and the note alone would be noise. * src/rqtl.rs, src/gemma.rs (peak_marker): strongest LOD / smallest p_lrt, with unit tests for both shapes and the empty cases. * src/routes/{rqtl,gemma}.rs: the `next` link and the plot note, when the request named a group. * docs/www/ai-instruction.md: 4b rewritten around what a reader does with a PheWAS -- where the position comes from (a mapping's `next`, a trait's `locus`, or chr+mb), the parameter table, what a hit contains, and how to read it (the 460 LRS cap, `traits: []` versus `markers: []`, peak-only and published-only limits, several traits on one marker), with the chr 9 example; 4a points at the new forward.
11 daysphewas: every trait peaking within a window of one positionPjotr
PublishXRef records the marker each published trait's QTL peaks at (Locus) with its LRS, which is a PheWAS waiting to happen: fix a position, find the markers within a window of it, and ask which traits peak at those markers. GET /groups/{group}/phewas?locus=rs30524230&window=10000 GET /groups/{group}/phewas?chr=9&mb=29.940961&window=10000 The window is in base pairs, centred, default 10 kb either side -- the "10K up and downstream" case. Positions come from the group's genotype marker table (the endpoint added last round), so `genotype_version` selects whose markers define the window and the answer reports them alongside the traits: an empty `traits` list is then readable as "nothing peaks that close" rather than a failure, and a ten-times wider window is offered as `next`. Measured on real data, ±10 kb around rs30524230 (chr 9, 29.94 Mb, a BXD QTL peak at LRS 460): 4 markers, 10 traits, from LOD 99.9 down to 2.68 -- ANPEP_band, then microbiome/haematology/infectious-disease traits. That cluster of 99.9s is the 460 LRS cap, not a spectrum, which the notes say. * src/db.rs (get_phewas_traits): PublishXRef rows for a set of peak markers, joining Phenotype/Publication through the existing trait_info_sql, strongest first; marker names are re-validated before they reach the IN list. * src/routes/phewas.rs: the endpoint, token-gated; window/centre validation (422 for none or negative), 404 for an unknown locus, notes for the empty cases, and a unit test for the version/group helper. * tests/phewas.rs: markers in the window, traits sorted by LRS with lod = lrs/4.604, every hit peaking at a listed marker, zero-width window, the two failure modes, and that the suggested wider window resolves. * README.md, docs/www/ai-instruction.md (new 4b: how to read a PheWAS), docs/www/operator-checklist.md: the endpoint and its reading.
11 daysversion: no notes array, so the envelope's empty case stays meaningfulPjotr
*tests/envelope.rs* uses /version as its example of a response whose `notes` is an empty array; adding a note there broke that check for no gain -- the fields document themselves, and tests/version.rs asserts their shape. * src/lib.rs (version): drop the with_note, with a comment saying why.
11 daysversion: stamp the commit the binary was built from, and prove itPjotr
/version reported 4831e5a while HEAD had moved on to 18d18b1: the stamp is written at build time and build.rs only declared .git/HEAD, which keeps saying "ref: refs/heads/main" across commits, so a rebuild after a commit reused the old hash. It now declares the ref file and the index as well, which is what actually makes a rebuild re-stamp. Three additions make the stamp trustworthy rather than merely newer: - `-dirty` when the working tree had uncommitted changes at build time, so a binary built pre-commit can never claim to be that commit; - `source_digest`, a hash over the compiled source files, which tells apart two builds of the same commit -- or of the same dirty tree -- where a commit hash cannot; - `branch`, so a detached HEAD is visible. * build.rs: watch .git/HEAD, the ref it points at, and .git/index; stamp dirty, digest and branch as well as the hash and commit date. * src/lib.rs (VersionInfo): the three new fields, plus a note saying a stamp is never advanced by committing without rebuilding.
11 daysgenotypes: expose the metadata, never the genotype callsPjotr
The listing named versions but not files; a caller could pick one and still not see what was in it. Three token-gated routes now describe a version completely, short of the matrix: GET /groups/{group}/genotypes versions, files, sizes, current GET /groups/{group}/genotypes/{version} file name, header labels, individuals, strains, chromosomes GET /groups/{group}/genotypes/{version}/markers markers (name, chr, cM, Mb), paged The detail reads the file's own `@`-labels (type, mat, pat, het, unk, scale) and its panel in column order, and reports both the whole panel and the strains a mapping uses after the parents/F1s rule -- so a caller can check a request before sending it and interpret a result after. Markers come back as positional rows with `meta.columns`, paged with `?page`/`?limit` and filterable with `?chr`, following /compute/gemma's shape rather than inventing one. The genotype calls stay on the server: file_meta() reads the matrix to skip it, and nothing returns a cell. * src/genotypes.rs (Marker, FileMeta, file_meta; GenotypeFile::file): header labels, individuals and marker positions; unit test. * src/routes/genotypes.rs: the two new handlers, token-gated like the listing, plus `file` in the listing and counts in the detail. * tests/genotypes.rs: the listing carries file names but no paths. * README.md, docs/genotypes.md, docs/www/ai-instruction.md: what the three routes give, and that the matrix is not among them.