about summary refs log tree commit diff
path: root/src/routes
AgeCommit message (Collapse)Author
45 hourscorrelation matrix: clustered heatmap and its plot HEAD mainPjotr Prins
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix reply and `POST /plot/heatmap` draws it: the r matrix in that order, the dendrogram that produced it, and a diverging scale with blue negative and red positive. A reader can see *why* two traits are neighbours instead of trusting an ordering. Clustering decisions, all recorded because an unrepeatable clustering is worse than none: * distance `1 - r` -- the usual choice for correlations, and it keeps anti-correlated traits apart, which is what a heatmap of correlations should show; * **average linkage (UPGMA)** -- single linkage chains along a gradient and complete linkage makes one flat blob out of noisy traits; average is the compromise GN1's R heatmaps used too; * **deterministic ties and leaf order** (lower index first), so two runs on one matrix produce the same picture and plots stay diffable; * a matrix with a **gap is refused** rather than clustered: an unknown distance treated as 1 would invent a structure, and the reply's note says to drop those traits and ask again. Pairing with the PCA work this is the visual half of a collection: scree plot for how much structure there is, clustered heatmap for where it is. Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back. Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
46 hoursplot: the scree plot of a matrix PCA, which GN2 draws client-sidePjotr Prins
GN2 renders a scree plot with Plotly in the browser, from the explained variance ratios its matrix page computed. This is the same figure as a file: `POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns image/svg+xml, like every other plot endpoint -- the plot is of the numbers that were computed, nothing is recomputed. scripts/plot-pca.py draws a bar per component (its share of the variance, GN2's number), a cumulative line with the 80%-of-variance crossing annotated (what a reader actually uses to choose how many components to keep), the eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps in the matrix -- the one thing GN2 hides by mutating its shared-sample list. `kind=N` on the request caps how many components are drawn, since a 200-trait matrix would otherwise draw 200 bars. Also: the monitor's SVG judge now understands a binary reply (the client hands back content type and size, not the bytes), and `plot-pca` is a check of its own: it asks the matrix for a PCA and posts the reply to the plotter. Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative path, which only resolves when the service's working directory is the checkout). plot-pca and plot-genotypes both ok.
46 hourscorrelation matrix: PCA, behind its parity testPjotr Prins
`"pca": true` on the matrix request now folds the correlation matrix into principal components: eigenvalues descending, explained variance ratios, loadings per kept trait, and the eigenvalues' sum reported so a caller can check the decomposition is sane instead of trusting it. The eigendecomposition is cyclic Jacobi -- symmetric matrices, no LAPACK, no new dependency -- capped at 300 traits because Jacobi is O(n^3) per sweep and this is for a collection, not a dataset. Two decisions from ai-memory #103, both visible in the reply: * **Traits with any null cell are dropped and named**, with the reason. A component from a matrix with holes is not defined; GN2 silently mutates its shared-sample list instead, and nobody can reproduce that component. * **The sign is fixed**: within a component the largest absolute loading is positive. Eigenvectors are defined up to sign, so without a rule two correct implementations disagree and no parity test could be exact. Verified against numpy.linalg.eigh on the same r the reply carries (scripts/pca-parity.py, BXDPublish 10001/10002/10003/10005/10007): eigenvalues: worst relative difference 3.2e-14 loadings: worst absolute difference 9.99e-16 trace: 5.000000000000 for 5 traits Unit tests cover the identity (unit eigenvalues), the closed form for 2x2 (1 +/- rho), the trace, a non-symmetric matrix being refused, and the cap. The monitor gained `compute-correlation-pca`, whose judge checks that the eigenvalues sum to the trait count and the ratios to 1.
3 daysgemma: suggest BulkLMM rotations for a threshold, with the numbersPjotr
GEMMA's reply offered R/qtl as the way to get significance thresholds, which is the wrong advice for a relatedness-structured panel: haley-knott's permutation shuffles phenotypes without regard to the kinship, so its null is anti-conservative exactly where an LMM was chosen in the first place. BulkLMM is the same model family as this scan and permutes by rotating the data through the kinship decomposition, so its thresholds do respect relatedness -- and it is fast. The 200 reply now carries three actions in order: draw it (/plot/gemma), POST /compute/bulklmm with a ready-to-post body (group, trait_name, samples, nperm 500, seed 1), then POST /compute/rqtl relabelled as a non-LMM cross-check with a warning that its threshold is anti-conservative here. Measured on BXD trait 10001, so the text asserts nothing it has not seen: GEMMA's best marker is rsm10000005700 on chr8 (p_lrt 5.9e-07, -log10 p 6.23) and BulkLMM's best is the same marker (LOD 3.866, beta -0.0615) -- at the peak the two LMM families agree on the locus. Their thresholds are not interchangeable: with 500 rotations BulkLMM's 0.05 threshold is LOD 3.78 (the peak clears it, and a second seed gave 3.62), while R/qtl2's LOCO scan1perm on the same trait and permutation count gives 2.62, so a global kinship is the stricter null; and LOD 3.87 and -log10 p 6.23 are different scales for one marker, so a BulkLMM threshold must never be compared with a GEMMA p-value directly. The 422 permutations_not_supported text quotes the same measurements instead of prose, and the suggested body was run verbatim to check it: POST /compute/bulklmm with it returns 200, 21056 rows, and the threshold note "500 rotations (seed 1): alpha 0.01 LOD 4.10, alpha 0.05 LOD 3.62".
3 daysqtl2: permutations for the LOD thresholdsPjotr
POST /compute/qtl2 gains nperm, perm_seed and perm_strata, and the reply gains meta.permutation {n, seed, significant, suggestive, values} -- the genome-wide maximum of every permutation, so a histogram needs no second call. `data` stays the positional rows array on purpose: /plot/rqtl posts it straight back and the monitor counts its rows, so the null went beside the rows rather than around them (a generic Envelope::with_meta/Meta::extra, skip_serializing_if, so no other endpoint's response changes). Thresholds come from R/qtl2's scan1perm over the same genotype probabilities, kinship and covariates as the scan, with chr_lengths from the map -- without those it derives chromosome lengths from marker positions and the numbers stop being comparable with R/qtl's. The permutation CSV is written in R/qtl's two-column shape so one parser and one percentile routine serve both engines: PermutationResult::from_values is now shared with /compute/rqtl instead of duplicated. Semantics, each with the note that says it: * scan1perm shuffles phenotypes across individuals -- right for an outbred cross, only approximate for a selfing RIL panel, where qtl2perm is the correct package (not installed here, and not packaged); * perm_strata takes a case attribute name (sex, Epoch, ...) and permutes within it, because shuffling across sexes inflates the null; * fewer than 100 permutations gets a note that the 0.05 quantile is a quantile of too few values; more than 10 000 is refused, since each permutation is a whole genome scan; * a seed is reported always -- without one, two identical requests would give different thresholds. Cost, measured on BXD trait 10001 (34 phenotyped, 20 chromosomes, LOCO kinship, 4 cores): 20 permutations 0.6 s, 100 in 2.2 s, 500 in 14 s, 1000 in 24 s through the API; 0.022-0.03 s per permutation. Thresholds converge as expected (0.05: 4.331 at 100 and 200 permutations, 4.374 at 1000 on the raw probe) and are reproducible: the same seed gives the same numbers on 1 and 4 cores. Four things that only surfaced by running it, all commented where they bit: optparse keeps the dash (--perm-out is opt[["perm-out"]], so opt$perm_out was always NULL and the file landed somewhere the caller did not look); perm_strata must be a named character vector, not a factor; the LOD in the CSV is the LAST numeric field (R/qtl's shape puts the permutation index first, and a first-field parser reads that as the LOD); and percentiles interpolate at (n-1)p/100, so 100 maxima give 0.6733, not 0.67. bin/monitor-gn4 gains compute-qtl2-perm (nperm 10, judging meta.permutation) so a deploy that loses the R path shows up as a red check rather than a surprise.
3 dayscorrelation: the matrix endpoint, and the rule that a deploy ends with a ↵Pjotr Prins
monitor run `POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}` references -- a collection, as GN2's matrix form posts it, which spans datasets -- and answers an N x N matrix of r, p and n. The five decisions that are not obvious, each recorded where a reader will look for them: * **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to Spearman mid-row once a Pearson r passes 0.999, which the self-pair always does, so its matrix is asymmetric by construction. Here the method is chosen once and r[i][j] == r[j][i] exactly. * **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine calls, not N^2. * **The diagonal is by construction**: r = 1, p = null, n = the trait's own count, so half the traits cannot come back as 1.0000000000000002 with a NaN p. * **Per-cell n is its own matrix**, and a cell under `min_overlap` (default 6) is null in r and p while its n still says how many points there were. GN2 accepts 2, where |r| = 1 by definition. * **The sample axis is the first trait's group** (#106), resolved once and stated in the reply with the number of individuals it kept. Fetches are one query per dataset (`get_dataset_values_for_traits`, `get_publish_values_for_traits`), not one per trait: a matrix chooses a handful out of twenty thousand. Capped at 1 000 traits, because the cost is quadratic in the correlations and in the reply. No PCA -- the r matrix has to be trusted on its own first. Verified twice: cells agree with scipy to the last bit (BXDPublish 10001/10002/10005: 0.6756270426459695, 0.6950450737755965, 0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964, 0.8819879006700073), and a cell equals what the row endpoint says for the same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix, `min_overlap` above every cell's n, an unknown trait, `bicor`, and no traits at all. The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier all`, and says why: `/version` can report a commit and a reachable database while every engine that shells out is dead. The monitor gained a matrix check whose judge enforces square, symmetric and a unit diagonal.
3 dayscorrelation: verify the partial axis against numpy/scipy, and clamp a perfect rPjotr Prins
The implementation had already landed in 698e6ab by accident -- a `git add -A` swept it in with the operator-checklist docs, which is why its message says nothing about partial correlation. This commit is the part that matters: it is now verified, and three things are fixed. **Verification.** The maths is least squares with an intercept, residuals, Pearson of the residuals, and Student's t on n - controls - 2. Checked two ways against independent code: * BXDPublish 10001 with 10002 held constant, target 17164, n=6: ours r=0.9960864631522379 p=0.00029371957431469716, numpy lstsq + scipy.stats.t.sf r=0.9960864631522379 p=0.00029372. Zero-order: ours -0.10577609106505677, scipy pearsonr -0.1057760910650567. * The same for 17084: identical to 16 digits, r=0.9840718340719924. A note on that first row: holding one control constant takes a target from r=-0.11 to r=+0.996 on six points. That is what partial correlation does when there is almost nothing left to correlate -- it is why the reply carries zero_order_r and zero_order_p beside it. **Fixes.** `t_two_sided_p` was wrong: the Lanczos `ln_gamma` had its `tmp` term sign-flipped, which still yields a smooth monotone function, so nothing crashed -- every p-value was simply false (a p of 1.8e-24 came out as 1.0). The corrected version matches `scipy.stats.t.sf` to 1.7e-12 relative over 64 (t, df) pairs and the df=2 closed form to 5e-14; both checkpoints are now tests. A perfect partial r (self-correlation) is clamped to +-1 with p=0.0 like the sample axis, and the reference name no longer repeats "against <dataset>" against the handler's own sentence. The monitor now exercises the axis: `compute-correlation-partial`.
3 daysoperator checklist: the tool block env.sh must keep, and why BulkLMM cannot ↵Pjotr Prins
work yet Writing down what the monitor found and what fixed it, because it is config that lives outside the repository and is therefore easy to lose at the next deploy: with only the cache dirs set, every engine that shells out answers 502 while the whole read surface and search look healthy. Measured on production after the fix (bin/monitor-gn4 --tier compute): 8 ok, 1 failed -- correlation, gemma (21 055 rows), rqtl, qtl2 (20 453), fitqtl, ravanan (21 055 rows), bnw, bnw-examples. Before: 1 ok, 7 failed, 1 busy. The remaining failure is BulkLMM and it is not a config line: the Julia project is registry-based and no depot on this box has its packages, so it stops at "Run Pkg.instantiate()". Populating a depot downloads from the Julia registry, which should be a decision rather than a side effect.
3 dayscorrelation: tissue and literature axes, paired the way the data actually isPjotr Prins
`type` on /compute/correlation now selects GN2's `corr_type`: sample (default), tissue or literature. Partial is refused by name rather than silently treated as sample. **tissue** reduces each gene symbol to its highest-Mean probe in TissueProbeSetXRef and pairs the two value vectors **by TissueID**. That last word is the difference from GN2, which keeps each vector as a bare list and pairs by position on the assumption that every probe covers the same tissues. On this database it does not: every probe carries exactly 26 values (liver, kidney, spleen, ... peripheral) or exactly 32 (striatum, hippocampus, neocortex, ... brain regions), the two sets are disjoint, and 45 281 probes are in each. Pairing those positionally correlates liver expression with striatum expression because both happen to be entry one. Pairing by tissue means a cross-panel pair has no correlation at all, which is the honest answer and makes the panel boundary visible instead of inventing a number. Verified independently for Shh vs Hcrtr1: 0.7222718302270719 from the endpoint, 0.7222719072696666 from a separate Pearson over the same TissueID-aligned pairs (the residue is MySQL's single-precision `float` column as printed by the client). **literature** looks the pair up in LCorrRamin3 in both directions after translating rat/human ids through GeneIDXRef, in chunks of 4 000 genes (the first attempt died on MySQL's 65 535-placeholder limit). It is a precomputed value, not a statistic: p and n are null. A trait with no gene id or no row is left out and counted in the note, where GN2 reports 0.0 and fills the tail of its top-N with them -- that reads as "measured, no relationship", which is not what happened. Gene ids come from ProbeSet.GeneId, not ProbeSetXRef.GeneId: the latter has no such column here, which GN3's own mapping also records.
3 dayscorrelation: run over the group's sample axis, and report a perfect ↵Pjotr Prins
correlation as 0.0 Two divergences from the live genenetwork.org, both measured against it. **The sample axis.** GN2's `get_sample_r_and_p_values` walks `target_dataset.group.samplelist` and takes the reference value by name: the correlation runs over the group's animals, not over every individual the trait happens to have a value for. Trait 1427571_at of HC_M2_0606_P carries 99 values, the BXD panel holds 67 of those individuals, and the answer changes materially -- GN2's top hit is a -0.845 probe that does not appear in our top five at all. `corr_samples_group` now takes `samples_primary` (default, the group's genotype panel, which is where GN2's list comes from) or `samples_all` (the old behaviour); the panel is read with `genotypes::resolve`, falling back to the group's used-for-mapping strains where a group has no genotype file on this machine. `samples_other` is refused rather than guessed. **A perfect correlation.** The engine's r can land at 1.0000000000000002, which makes 1 - r^2 negative and its p-value NaN. The live API answers 0.0 for that row (scipy's pearsonr), so the reply now clamps: r to +-1 and p to 0.0. That is the reference's *output* being reproduced, not a computation being redone -- #96 stays resolved in the same direction. With the axis fixed, n agrees with the live site exactly (67) and r agrees with GN2's own published per-sample values to 1.4e-8. What does not yet agree is GN2's reported r itself: over the same 67 individuals, its `/correlation` endpoint says -0.8446202819617522 for 1447043_at while correlating the values its own `/sample_data` endpoint publishes for the same two traits gives -0.8393085803732605; on a second pair the gap is -0.8479321530363810 against -0.8451666639589470. Both pairs are off in the same direction and by a similar amount, so the two GN2 endpoints are reading different values for the same samples -- not a difference of sample sets (all four BXD genotype files give the same 67) and not a transform (2**v, 10**v, exp, v**2 and log were all tried). Until that is settled, our numbers are the reference's published values correlated correctly, and the ranking cannot be identical to a number we cannot reproduce from the reference's own data.
3 dayscorrelation: phenotype traits and datasets, the common casePjotr Prins
Either side of a correlation may now be a Publish (phenotype) dataset, not just a ProbeSet one: a pheno trait correlated against an expression dataset is what GN2's mapping page is usually used for, and returning 404 for BXDPublish was the wrong answer. A publish reference trait is addressed by its numeric PublishXRef.Id -- that is how GN2 addresses phenotype traits, and a name there is refused with a message saying so rather than a confusing "no such trait". The bulk fetch gains the publish equivalent, keyed on the freeze's InbredSetId, so the same individuals line up on both sides. Measured live: BXDPublish/10001 (34 observations) against HC_M2_0606_P, top 5, HTTP 200 in 5.0 s; 1440079_at at r -0.757 (p 3.2e-6) leads.
4 dayscorrelation: the sample-axis endpoint, synchronousPjotr Prins
POST /compute/correlation now computes instead of returning canned data: one reference trait against every trait of a target dataset, observations being individuals, pearson or spearman, top-N by |r|. The reply is positional -- trait, r, p, n (column names in meta.columns) -- with the method, the minimum overlap and the top-N stated in the notes. `bicor` is refused with the methods that exist; a bad top_n is refused rather than clamped; a dataset or trait with nothing to correlate says so instead of returning an empty list. src/db.rs gains the bulk fetch the sample axis needs: every value of every trait of a dataset restricted to the reference trait's individuals, one record per (trait, individual, value), ordered by trait so the caller groups as it streams. It never loads a dataset whole -- ProbeSetData is 5.3e9 rows, so the query narrows by dataset and by individuals, which is also why the sweep, not the arithmetic, is the whole cost. Measured on the live db_webqtl: HC_M2_0606_P / 1427571_at against its own dataset, top 5, HTTP 200 in 16 s; the trait correlates with itself at 1.0 over 99 observations, then 0.802 (p 1.9e-23), 0.790, -0.789, -0.788. Synchronous is affordable at this size; GN2's job indirection exists for top-N 20000 across datasets, which is the next thing to time. Known and deliberate for now: the publish/genotype axes are not wired (the fetch is probeset-only), and a perfect correlation reports p as null -- GSL's t statistic is undefined at |r| = 1, whose r comes out as 1.0000000000000002, so 1 - r^2 is slightly negative and the engine returns NaN. Reproducing the engine exactly, rather than "fixing" the p-value here, is the decision recorded in docs/correlations.md.
4 daystokens: short opaque API tokens in a single tokens.sqlitePjotr Prins
A second credential format beside gn-auth's JWT, for agents and scripts: `gn4ai_` + 56 body characters + 2 checksum characters = 64, from an alphabet with no look-alikes, so a typo is refused before the registry is consulted. Opaque by design -- nothing is stored in the token; every fact lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode 0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row, takes effect on the next request, and a token in a log line reveals nothing; lose the registry and every short token stops working. Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and GET /tokens/who are behind a router layer that refuses any caller whose TCP peer is not 127.x.x.x/::1, and any request claiming to come through a proxy. A layer rather than a per-handler check, so nothing of a remote request is parsed before it is refused; there is no configuration switch, because minting credentials must never be reachable from the network. A short token reads public data only: every token-only endpoint, and `#[gnauth(secured)]` endpoints for public rows; a confidential dataset is refused with a 403 that names the JWT route, since gn-auth cannot validate an opaque token (the on-the-fly-bridge and registry-grant alternatives were both rejected for v1). scripts/gn4-token.py create|list|revoke|who drives the routes, so there is one implementation of the format and the revocation rules; the token is printed exactly once. Tests: the localhost guard over all four routes (remote peer, no peer, proxied header), and mint -> authenticate -> who -> revoke -> 401.
4 daysgroups: the design in /groups/{group}, and a drift check against GN2's ParInfoPjotr
Two follow-ups to the parental-design endpoint. /groups/{group} now carries the headline in its notes -- "Parental design: C57BL/6J (maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes the parents and F1s by default" -- plus a next link to the full design. "Which cross is this?" is two strain names, and needing a second request to see them was the only reason the endpoint had to be called first. A group with no metadata says so in the same note, pointing at parents_and_f1s.json and at `individuals.exclude`, rather than leaving the reader to wonder why nothing was dropped. The metadata is the same fact in two places (GN2's ParInfo, our JSON) and it drifted once already, silently: the paternal/maternal labels were swapped against GN2's unpacking, and nothing noticed because the exclusion rule takes both strains regardless of role. scripts/check-parental-metadata.py parses ParInfo with ast (no gn2 import) and compares it group by group, reporting disagreements per role and treating groups the JSON knows but GN2 does not as informational. tests/parental_metadata.rs runs it while a GN2 checkout is present (GN2_CHECKOUT, or a sibling path) and skips once GN2 is retired -- at which point the JSON is the only copy and there is nothing to drift from. It reports 23 groups reproduced, 1 extra (BXD-Longevity).
4 daysgroups: an endpoint for a family's parental designPjotr
GET /groups/{group}/parents-and-f1s answers the question the mapping notes only implied. It reports the F1 strains, the maternal and paternal strain (from the deployment's parents_and_f1s.json, ported from GN2's ParInfo -- all 24 groups), the @type of the group's genotype file, the individuals each role names where the metadata names them (BXD-Longevity does, most do not), and -- the part a caller needs -- which members of the current panel the default rule drops and which the `F1` name heuristic catches. Two things that came out of building it and comparison with GN2: * the JSON's paternal/maternal labels were INVERTED against GN2's reading. GN2 unpacks ParInfo as `f1, f12, maternal, paternal`, so BXD's maternal strain is C57BL/6J; our file said paternal. No mapping was affected (non_strains takes both regardless of role) but the metadata was wrong, and anything using "maternal" later -- reciprocal F1 direction, the @mat/@pat note -- would have been backwards. The file is rewritten from ParInfo itself (ast-parsed, not retyped), 23 groups' roles filled in and the 24th (BXD-Longevity, which GN2 does not have) left as it was. * an RI set has two kinds of F1 name: the design's (B6D2F1) and the panel's own intercross individuals (C57BL/6JxBXD037F1). The endpoint reports excluded_from_panel and excluded_by_name separately and says which is doing the work, because "0 declared, 37 by name" is otherwise confusing. Empty F1 names (C57BL-6JxC57BL-6NJF2 has two) are dropped rather than becoming exclusion entries that match nothing. genotypes::non_strains is rebuilt on the same typed parser (DesignRole / ParentalDesign), so the rule and the endpoint cannot disagree. tests: a unit test for the roles, empty-name handling and the unlisted-group source, and an integration test asserting BXD's maternal/paternal orientation and the note.
4 daysindividuals: select who enters a mapping, by namePjotr
Parents and F1s are excluded by default (as in GeneNetwork2/3), with include_parents_f1 to keep them all. That boolean was not enough for what was asked -- "include these, drop that substrain" -- and the exclusion list itself is data (parents_and_f1s.json, seven groups) that no request can reach. F1 names also differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH, C57BL/6JxBXD037F1 for the intercross F1s), so pattern matching is the wrong tool. Every group-mapping endpoint -- /compute/gemma, /compute/gemma/ravanan, /compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm -- now takes "individuals": {"include": [...], "exclude": [...]} applied after the parent/F1 rule, exclude winning over include. An unknown name is a 422 invalid_individuals whose message lists the panel: a silently dropped animal changes a kinship matrix without anyone noticing, so the failure has to be loud. genotypes::select_panel is the one implementation, and it returns what it did (dropped parents/F1s, re-included by name, excluded by name) so the Genotype note can say it: "197 individuals, 38 parents/F1s dropped; 1 excluded by name". The note now also spells out the distinction that makes this option treacherous: the kinship uses *every* individual in the genotype file, the association uses only those with a trait value -- GEMMA reads no phenotypes for the kinship. So "include the F1s" changes the relatedness structure without putting the F1s in the fit, and a request can legitimately have 234 individuals in the genotype file and 34 in the association. Verified on BXD.8: excluding BXD1 gives 197 individuals / 33 values; include_parents_f1 with an F1 re-included and a substrain excluded gives 234 / 34. A basket supplies values by name, so a selection applies to a basket-fed mapping the same way: the individuals excluded are simply not in the panel any more, and their basket values are ignored. GenotypeError gains Invalid for this (422) so a caller's typo is not reported as a missing genotype file; the read-only endpoints that can never see one map it the same way for exhaustiveness.
5 daysbnw: module scores as trait values, and BNW's worked examplesPjotr
Two additions that make the endpoint usable rather than merely correct. POST /compute/bnw/features -- from a network to a mappable trait. Every node with at least one parent at the threshold is a module (the node plus its parents), scored per sample by the first principal component of the module's standardised members, sign-corrected so the score rises with the node. Without ?module= the reply is one row per module -- module, members, size, variance_explained -- where variance_explained says whether the score is a fair summary at all. With ?module=NAME it is that module's per-sample scores in the {name, value} shape every mapping endpoint already takes, so the chain runs network -> module -> /compute/rqtl or /compute/gemma with no translation layer. That is what BNW is for in GeneNetwork; the diagram is a by-product. The notes name the construction (a citable module score must say what it is) and repeat the caveat: a module is a correlation structure, not a QTL. GET /compute/bnw/examples and /compute/bnw/examples/{name} -- BNW's own worked examples, for an agent to explore instead of inventing data. The five datasets its tutorial and examples pages present are listed with their sizes, and each is served as a matrix in the values shape /compute/bnw takes, with a ready-to-post action carrying the body. ?columns= restricts the matrix (the spleen example has 14 variables and a caller may want fewer). genotype_column warns that a column is a genotype code: this driver is continuous-only, so it enters the network as a number, the way the web application's GeneNetwork path treats it. Files come from BNW_EXAMPLES_DIR (a gn-bnw checkout's sourcecodes/ example_datasets); a deployment without it answers 404 no_examples. Measured on the dev instance: features over five BXD traits gives two modules (BXD::10001 with BXD::10003 and BXD::10041, 60.4% of variance; BXD::10044 with BXD::10001, 27.2%), and BXD::10001's 19 scores posted straight to /compute/rqtl and /compute/gemma both return 200 -- gemma peaks on chromosome 6. tests/bnw.rs covers the module list, one module's scores in mapping shape, the unknown-module 422, the example list, one example's matrix and its ready-to-post action, and a 404 for a name that does not exist.
5 daysplot/bnw: two SVG figures, drawn the way /plot/gemma draws onePjotr
Post the reply from /compute/bnw back and get an SVG: the same plumbing as /plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/ svg+xml, the plotter's line in x-plot-summary), nothing recomputed. Two figures, because one cannot say what the other can: * the network (default): nodes on a circle, one arrow per edge that survived the threshold, arrow width and label = the edge's posterior, and the below-threshold edges as faint dotted lines -- so a reader sees what was *considered* as well as what survived, which is half of why the posterior table exists; * ?kind=posterior: every candidate edge as a cell in a matrix, dark = high posterior, outlined cells are the ones above the threshold. The diagram cannot show that an edge came within 0.02 of the threshold; this can. Pure-Python SVG, standard library only: no graphviz in the deployment. A circular layout is less pretty than a force-directed one and is deterministic, which matters more for a figure that accompanies a JSON result. Node labels keep the distinguishing tail of a long id, with the full name beside the node. Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the second figure. tests/plot.rs covers both figures plus the 422s (not a BNW result, unknown kind).
6 dayscompute/bnw: a Bayesian network over traits, called out to BNW's driverPjotr
POST /compute/bnw learns a network over a set of variables and returns one row per candidate edge: from, to, posterior, in_structure. Inputs are published traits of a group (group + traits) or an explicit matrix (values: one entry per variable with per-sample values), which covers the case the database cannot -- a basket's columns, a handful of probe sets. BNW's knowledge stays in the gn-bnw repository. GN4 writes a variables x samples matrix, runs `bnw learn` (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR) and parses the reply: nothing about the three C programs, the private work directory or the type line in the input file lives here, so BNW's pipeline can change without a GN4 release. The driver's failures map onto the API's own: bad input 422, a missing driver 502 bnw_not_installed (its exit 77), a failed computation 502 bnw_failed in the driver's words. Decisions worth recording: * parameters are BNW's own defaults -- maxparents 4, k 1, tiers 3, threshold 0.5, read from its create_tiers_gom_part1.php rather than chosen here -- and the reply names the values used, because a network at 0.5 and one at 0.9 are different answers to the same question; * at most 40 variables and at least 3 samples complete across every variable. The first is the structure search's exponential in-degree; the second is BNW modelling a complete multivariate normal, so a sample missing any value is dropped -- and the note says how many survived rather than silently shrinking the panel; * its own gate, always claimed (threshold 0): there is no such thing as a small BNW run, the cost is the search, not the number of individuals; * the notes say plainly that a posterior is not a p-value and that a network is not a QTL, because that is the easiest thing to over-read. tests/bnw.rs compares the endpoint against the answer the BNW *web application* gives for its own six-trait example (the same fixture as the driver's parity check, gn-bnw/test/golden/): the five edges must match, and the notes must name the parameters and carry the caveat. It skips when the driver or its binaries are absent. Not deployed to production.
6 daysgemma/ravanan: resolve the same inputs as the direct runner; refuse what it ↵Pjotr
cannot do Two gaps, both invisible until a real request ran against a cluster. The endpoint read only genotype_file, snps_file and trait_values from the request; group, genotype_version and samples were accepted by the shared request type and silently dropped. A /compute/gemma-shaped request (group + samples) therefore arrived as empty paths and empty values and died with "I/O error: No such file or directory (os error 2)" from reading "". Running it exposed the second gap in the same breath: the gate sized the request from trait_values.len(), so a samples-only request was sized 0 and took no concurrency slot at all. resolve_inputs is now one function in routes/gemma.rs, called by both handlers: upload materialised into the pair the engine reads, basket resolved to named values, group prepared with the parent/F1 and empty-genotype rules, named samples aligned by name onto whatever individuals the file has, positional values checked against the panel size. That block was ~250 lines of duplicated rules inside compute_gemma, which is how it drifted in the first place; the direct runner now calls the same function and keeps only its covariates, normalization and gate. The ravanan reply also carries the input notes now (which genotype version, how many values aligned, what was dropped), so a run says what it mapped as well as where the peak is. Refused rather than ignored, because the CWL workflow has no step for either: covariates (422 covariates_not_supported) and normalize (422 normalization_not_supported), each naming /compute/gemma as the endpoint that can do it. A new test asserts both refusals happen before any job is submitted, so it needs no ravanan, ccwl or cluster. Verified against the Slurm container from gn-machines: group: BXD became 40 Slurm jobs (20 kinship + 20 GWA, scattered over the snps file's chromosomes), all completed, 21 055 rows in ~3.5 min. README and docs/ravanan-slurm.md record the four requirements that run turned up, three of them not obvious: ccwl on PATH, a ravanan build with the --slurm-api-endpoint fix, the *server's* HOME writable by the job user inside the cluster (gemma-wrapper locks $HOME/.<hash>.lck there), and GUIX_PACKAGE_PATH for the manifest.
7 daysnext: every link a client is offered is a call it can makePjotr
tests/envelope.rs follows every href in every reply, and running it end to end found six that were not addresses: - /species, /species/{name}/datasets and /groups/{group}/datasets offered /species/{species} and /datasets/{dataset} -- templates, which 404 when followed. Each now points at a real row from the listing in hand, and the description says any other name can go in its place. - /datasets/{name} offered /datasets/{dataset}/probesets/{probeset}: the listing it already links is the callable one, so the shape moved into that link's description. - /traits/{dataset}/{trait}/values offered a GET /traits/{ds}/{trait}/correlation, which does not exist: correlation is POST /compute/correlation and that reply already carries it in "actions". Removed, with the reason written where it was. - the same reply built its /groups/... links from the *dataset* name. BXDPublish's traits live in group BXD, and /groups/BXDPublish/phewas is a 404 (no genotype files for it). db::get_group_name resolves the group from InbredSet.Id and the links use it; when a dataset has no group the group links are simply absent. - /groups/{g}/phewas with no parameters is a 422 ("give either locus, or chr and mb"), so that link is now a shape with placeholders. Placeholders stay where the caller must supply a value (.../markers/{marker}/classes?trait={trait_id}), but the crawl now skips them only when the description says what to fill in -- otherwise a client reading the next list cannot tell a call from a template. Two stale expectations alongside: /version carries the root links now (it is the natural first call), and /search's results put its own follow-ups first and the root links after them. Tests are honest about their environment too: links skips when no search index is configured (an empty XAPIAN_DB_PATH answers 502), and the envelope crawl exempts /openapi.json, the API document, which is deliberately not enveloped.
7 daysgenes: the alias reply carries the entity's label and its gene link-outsPjotr
The GTEx problem -- it is human-only, and a mouse symbol there is an empty report -- is answered by the endpoint that already knows which entity it matched: /genes/{symbol}/aliases now returns label the entity's canonical symbol in that species ("Shh" for mouse, "SHH" for ?species=human) -- the symbol a human-only service needs links [{name, href, description}] built for that symbol and species: RGD, MGI, BioAlliance, NCBI Gene, Wikidata (the QID), UniProt, STRING, GeneMANIA and GeneCup, plus GTEx/Open Targets/Protein Atlas/GWAS Catalog for a human entity or BioGPS/Allen Brain Atlas for a mouse one So a mouse Shh reaches GTEx through the human ortholog's label rather than a guessed one, and a client no longer holds the URL vocabulary. Only links that answer are built: HomoloGene, UniGene, Genotation, GeneBridge and PhenoGen are gone, and Panther is browser-only (see SKILL.md "Gene link-outs", where each URL was checked). A cached entity written before the label was kept is refreshed rather than served -- otherwise the link-outs would be built on a symbol it does not have -- and `stale: true` still says when Wikidata was unreachable and the cached answer was used. tests/wikidata.rs: a mouse gene gets the mouse services and no GTEx; the human ortholog gets label SHH and a GTEx href built from it. Both halves skip when the reply is stale, since a Wikidata timeout is not a failure.
7 daysplot/bulklmm: the 0.05 and 0.01 lines, never 0.10 unless askedPjotr
Checked what the plot actually draws: it has never drawn the alpha 0.10 line -- the plotter matches 0.05 and 0.01 only. What lists all three is the reply's notes ("alpha 0.01: LOD 4.58, alpha 0.05: 3.65, alpha 0.1: 3.50"), which is where the 0.10 came from. Still, the point stands on its own: 0.10 is the loosest bar in the set and drawing it invites reading it as the line to clear. So the defaults are stated and pinned -- 0.05 and 0.01 on the plot, 0.10 only on request (?all_thresholds=true, or an explicit ?thresholds=) -- alongside GEMMA's Bonferroni, which is opt-in for the same reason: a reference line is an annotation, not a result. tests/bulklmm.rs counts the lines: exactly two by default, and no alpha 0.10 label.
7 daysdocs: say that /plot/gemma draws no line unless one is asked forPjotr
The OpenAPI description still promised "a Bonferroni reference line by default", and GEMMA's own suggested plot call said the same. Both now say what the code does, and where the number comes from: the plotter computes 0.05 over the analysed markers -- conservative under linkage and unrelated to relatedness -- and ?thresholds= takes one computed elsewhere, each line labelled with its source. Also corrected in the README's /plot section and in SKILL.md, where the same promise had been copied.
7 daysplot/gemma: no reference line by default; Bonferroni is one more labelled linePjotr
A line on a plot is an annotation, not part of the result, so the default is now no line at all. GEMMA's Bonferroni becomes opt-in like any other: ?bonferroni=true GEMMA's own line, labelled ?thresholds=3.65&threshold_labels=… another engine's, labelled (nothing) just the scan Which means a reader sees exactly the lines someone chose, each with its source in the label -- and the accidental "this plot says significant" impression of a default convention line is gone. The plotter's summary and its <desc> no longer mention Bonferroni unless it was drawn. tests/plot.rs: the plain plot has neither line; with thresholds the two BulkLMM lines are labelled and Bonferroni is absent; with ?bonferroni=true GEMMA's own line appears.
7 daysplot/gemma: draw thresholds computed elsewhere, labelled with their sourcePjotr
GEMMA has no permutation test here, so its plot carries a Bonferroni line. A threshold from /compute/bulklmm belongs to a different null and must not look like GEMMA's, so it is passed in and named: POST /plot/gemma?thresholds=3.50,3.65 &threshold_labels=bulklmm%20alpha%200.10,bulklmm%20alpha%200.05 Each is drawn dashed in the series colour with "label: value" beside the line, with GEMMA's Bonferroni line still there, and the plotter's x-plot-summary names the reference lines it drew. A threshold that is not a positive number is a 422 invalid_threshold rather than a silent omission. /compute/bulklmm's reply offers that href already, with its alpha 0.05 threshold filled in -- so "same values, two models, one picture" is one link from the scan, and the labels keep the two nulls apart. tests/plot.rs: the plain plot has no bulklmm line; with thresholds both are drawn and labelled; a non-numeric threshold is refused.
7 daysbulklmm: report the variance components, and say what a boundary meansPjotr
"is the kinship term doing anything?" should be answerable from the reply. The driver now prints h2 and the residual variance under the null model, and the reply carries both: Variance components: h2 = 0.000000, sigma2_e = 0.402171, estimated once under the null (intercept) model from y and K, not per marker. With a few dozen phenotyped individuals these estimates pile up at the boundaries 0 and 1 -- the likelihood is flat there -- so read h2 as "kinship matters a lot / not at all" rather than as a precise proportion. At a boundary (h2 ~ 0 the model is ordinary regression, which is why its LODs can match R/qtl's to the digit). Checked with a strain-ID-driven trait (values = rank of strain), which has h2 ~ 1 by construction: BulkLMM reports h2 = 1.000000, sigma2_e = 3e-06 and a peak of LOD 3.1 -- the kinship term absorbs the strain structure -- while R/qtl, which has no kinship term at all, reports a spurious peak of 5.06 on the same values. That is the discrimination the h2 column was missing.
7 daysbulklmm: BULKLMM_BLOCKS as a config option, default 4Pjotr
The marker blocks a single-trait scan is split into were the driver's default (Threads.nthreads()) and changed with --threads. They are a separate knob now: BULKLMM_THREADS Julia threads (--threads) default 4 BULKLMM_BLOCKS independent marker blocks, scanned default 4 in parallel (1 = serial scan) Blocks are exact either way -- LOD 9.589938 and the same permutation thresholds at 1 and 4 -- so the knob only buys wall clock. The reply says what ran ("split into 4 independent blocks and scanned on 4 Julia thread(s) (BULKLMM_BLOCKS / BULKLMM_THREADS ...)"), because the answer to "is it multicore?" should be in the response and not in the log.
7 daysgemma: the permutation refusal names both routes, including running GEMMA itselfPjotr
We do not run GEMMA permutations -- each repeats the whole LOCO scan and the gate gives GEMMA one slot -- but "not supported here" is not the same as "not available". The refusal now says both: 1. POST /compute/bulklmm with nperm: 1000 -- an LMM whose thresholds come from rotations that keep the kinship (R/qtl's nperm does not, and is anti-conservative on a structured panel); 2. run GEMMA's own permutation locally. The reply's notes already name the bimbam pair and the phenotype file, so `gemma-wrapper --permutate` or `gemma -lmm 9 --permutate` on those inputs reproduces exactly the null GEMMA would have used -- hours instead of seconds, which is the honest price of that particular null. docs/bulklmm.md gains the table this belongs in: three ways to get a threshold, ordered by how honest the null is for a structured panel (rotation with kinship; GEMMA's own LOCO permutation run locally; R/qtl's nperm as a sanity check only). SKILL.md says the same in two lines. tests/gemma.rs now asserts both halves of the refusal, and the earlier assertion on /compute/rqtl -- which stopped being true when the refusal was rewritten -- is replaced rather than left passing by accident.
7 daysbulklmm: exact marker-block parallelism, and what the cores are actually doingPjotr
Reported: a BulkLMM run does not look multicore. It is not a misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint calls -- has no `Threads.@threads` at all; the package threads `bulkscan()` (many traits) and bulkscan_helpers, not scan. The parallelism that was visible before was BLAS (OpenBLAS defaulting to all 96 cores here), which is why `user` time ran 4x `real` while Julia threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s. * The driver splits the markers into independent blocks and scans them on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact: the LOD for each marker is independent given the null variance components, which depend on y and K only. Verified identical -- peak LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks -- and permutation thresholds combine by taking each permutation's maximum across blocks before the quantiles, which is what a genome-wide threshold is. * The parsed matrix and the kinship are cached per panel (read 1.3 s -> 0.4 s, kinship 0.9 s -> 0), because both depend only on the cross. * The summary in `notes` now says what ran: julia threads, blocks, BLAS threads, and the read/scan split. * The hand parser is back: CSV.jl into a DataFrame is an order of magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s). * BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for one scan). Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks 18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of the run and each block re-estimates its variance components. Where the cores pay is bulkscan() -- many traits in one process, threaded over trait blocks -- which is the shape the next endpoint wants; documented in the README rather than left as folklore.
7 daysbulklmm: an LMM scan that can be asked for thresholdsPjotr
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values have no threshold we can compute: a GEMMA permutation repeats the whole scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual stand-in and it is the wrong null -- it shuffles phenotypes as if the individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global kinship matrix and permutes by rotating the data through the kinship's eigen decomposition, so a threshold costs seconds. POST /compute/bulklmm {group, genotype_version, samples|basket, nperm, seed, reml, assumption} Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8, 35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2 4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81, 0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's published locus and LOD, and a significant peak against a threshold this API could not compute before. Two differences from GEMMA are in every reply, not implied: a global kinship matrix rather than leave-one-chromosome-out, and h2_null saying how much relatedness was found (here ~0, so the fit is effectively fixed-effects -- which is why it agrees with R/qtl where GEMMA differs). * src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a strains x markers matrix (our BIMBAM pair is already the allele dosages BulkLMM wants), a marker map, the phenotype and covariates -- runs the driver through streaming::run_streaming_with_env, and reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid f64 in Rust and would otherwise be reported as a value. * scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds), with the seed recorded so a threshold set is reproducible. * gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is cheap, unlike R/qtl's. * Julia comes from the deployment's own store binary with a prepared depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request runs JULIA_PKG_OFFLINE=true, so a scan can never download anything. `--project=<path>`, not `--project <path>`: Julia 1.8 takes the space-separated form as a program argument. tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and seed are in the notes, and a scan without permutations says there is no threshold rather than leaving a bare LOD to be read as significant.
8 daysqtl2: name the genotype model, and say what the conversion droppedPjotr
Two engines reading one riset panel can fit different predictors, and the reply now says which one it used and what it cost -- with this panel's own numbers. * `genotype_model: "riself" | "f2"` on /compute/qtl2 (`het` kept as the older spelling: drop/keep). riself is the default: R/qtl's convert2riself coding, the H calls become missing and the HMM re-imputes them. f2 keeps them as a third genotype -- a diagnostic. * The conversion now counts the calls it drops and keeps the count beside the cross (conversion.json), so a reply can say "610684 heterozygous call(s) (12.7 %) were dropped". A cross converted before that file existed measures the count once, on first use, rather than never. * Crossed with the note that already explained the H drop, so the two read together: what riself does, how many calls it cost, and what f2 would give. Documented in the README as "Why R/qtl and R/qtl2 disagree on a riset panel", with the measurements (BXD trait 16627, BXD.8: R/qtl 9.59 at rs33299937; qtl2 riself 3.73; qtl2 f2 9.68), R/qtl's adaptive convert2riself rule quoted from its own source, and qtl2's side: riself has two states, f2 three, and `genril2` -- from qtl2 0.18, "the progeny are inbred lines", cross_info first column the number of generations -- is the honest model for residual heterozygosity and is available in the 0.42 we run. What is missing for it is recorded precisely: a founder-based cross expects individual genotypes coded as founder alleles rather than 1/2/3 (a prototype with numeric codes read every genotype as missing), so founder_geno from the two parental alleles and an allele-coded het remain to be done, with DOL as the counter-example (its H is a collapsed DO, not residual heterozygosity). tests/qtl2.rs: the model is reported (Cross: riself / Cross: f2), the dropped count is in the reply, keeping the calls finds more, and an unknown model is refused rather than ignored.
8 daysqtl2: an option to keep heterozygous calls, to test what dropping them costsPjotr
R/qtl's convert2riself drops the third genotype code (see the note below), and so does this converter -- but qtl2 then never sees the call, the HMM re-imputes it from flanking markers, and on a marker that is 14% H that is a different predictor from the one R/qtl fits. {"group":"BXD","genotype_version":"BXD.8","het":"keep"} -> crosstype "f2" On BXD trait 16627 (BXD.8, 35 phenotyped animals): marker Mb R/qtl qtl2 drop qtl2 keep rsm10000000889 5.652 9.59 3.79 9.98 rs33299937 5.653 9.59 3.73 9.68 rs13476332 6.311 9.59 3.70 ... rsm10000000890 6.312 9.59 3.67 ... So dropping the H calls is what costs qtl2 the peak: kept as a third genotype it recovers R/qtl's number (9.68 against 9.59). The option is a DIAGNOSTIC: an inbred line has no heterozygotes, so an f2-shaped cross is wrong for inference even when it reproduces the other engine -- it is how one finds out that the conversion, not the model, is the difference. The policy is part of the conversion cache key (v3), and the reply says when it is not the default. `het` applies only to a panel this server converts: a shipped control-file cross carries its own coding and is refused rather than silently ignored.
8 daystraits: genotypes at a confirmed peak, and the plot of themPjotr
Returning genotypes is the one thing this API otherwise never does, so it is fenced and the fence is the feature: GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker> * the server re-runs the R/qtl scan for the trait -- the same machinery and gate as /compute/rqtl -- and takes the peak from that run, rather than trusting a client that names a marker; * the marker must lie within ?window= Mb of that peak (default 10), or the answer is 422 not_the_peak quoting the peak's marker, chromosome, Mb and LOD; * the peak must reach ?min_lod= (default 3), or 422 not_a_hit; * only animals that have a value for the trait are listed, and only this marker's calls. Nothing else of the matrix leaves. On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals across A 110 / B 8 / H 74 -- the two a class of eight being exactly the thing worth seeing. POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's `animals` (or its `genotypes` map) with the trait's values and get an SVG with one point per animal, genotype class across, value up, class means marked. Nothing is recomputed -- the plot is of the numbers the client sends, like the other plots. New script scripts/plot-genotypes.py (stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key. tests/peakgenotypes.rs: the peak's calls for animals with values (with the counts agreeing with the rows), the SVG, and a marker on another chromosome refused with the peak quoted. Also keeps /compute/rqtl's error order: naming both a group and a file is "ambiguous" even when the request also forgot its values.
8 daysuploads: every engine maps an upload by name; same name replacesPjotr
Two things asked for, and one bug each. * POST /uploads/genotypes with an existing name now REPLACES it. A genotype is the user's file to keep, not a record the deployment curates; the digest in every reply says which version ran. * genotype_upload on all four engines -- /compute/{rqtl,gemma,qtl2, fitqtl} -- in place of genotype_file/group. The server writes the form each engine reads: a GN .geno for R/qtl and R/qtl2, a BIMBAM pair (new: genoupload::to_bimbam, coding from the base's @mat/@het/@pat, NA for missing, snps as marker/bp/chr) for GEMMA. No user path travels with a request. Verified against the full 3811-marker BXD.4 matrix uploaded as an update: rqtl 3811 rows, gemma 3809, qtl2 3730, fitqtl by marker. The bug worth naming: values given by name were being aligned against a file, and for GEMMA that file is BIMBAM, which has no individual names -- so the phenotype file came out empty and the wrapper failed with a Ruby backtrace that said nothing. GEMMA's values are now ordered by the upload's own individual list before the pair is written. The related fixes: rqtl and fitqtl align named values onto the file's individuals (they previously demanded its column order), and genotypes::align_to_file is the one place that does it. tests/genoupload.rs: mapping an upload on rqtl/qtl2/fitqtl with values by name, and replacing a name (one row in the listing, a new digest). GEMMA's LOCO kinship needs more markers per chromosome than a small fixture has, so it is exercised against a full panel rather than in the fixture.
8 daysuploads: map an uploaded genotype by name, and clean up after itPjotr
Three follow-ups from putting an upload through the whole loop. * /compute/qtl2's `genotype_file` branch only accepted positional `trait_values`, i.e. it demanded the file's column order -- which a caller with an uploaded population does not have. It now takes `samples` (name and value) or a `basket` and aligns them onto the cross by name, leaving individuals without a value as missing. Positional `trait_values` still work for a file whose order is known. * DELETE /uploads/genotypes/{name} now removes the .geno it wrote on demand: a file nobody can name is a file that confuses the next operator. * scripts/example-genotype-upload.py spreads its markers across chromosomes. One chromosome of markers is a poor example -- a leave-one-chromosome-out kinship has nothing to leave out, so the scan came back with no rows at all, which looked like a broken upload and was really a property of the example. 20 markers over the genome map cleanly (19 rows, the 20th used for the HMM), and the README says to use "kinship": "overall" if a file really has one chromosome.
8 daysuploads: a genotype matrix that updates an existing populationPjotr
Tab-delimited, header names the individuals from column 4, then marker, chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own SQLite file -- nothing goes near MariaDB -- and written out on demand as a GN .geno carrying the base's @-metadata, so the R scripts read exactly what was uploaded. An upload updates a population, it does not invent one, and the checks say so: * the base version must exist (resolved through the same genotypes::resolve_named the rest of the API uses); * at most 50 individuals the base does not have, and at least one that it does -- no overlap is a different population, not an update; * at most 200 000 markers, and the refusal says the sysadmin has to organize a file that size, because this is a request path, not an ingest; * the values must be the same kind as the base's: letters, whole numbers or probabilities. A letters panel holding 0/1 would be read as the wrong thing silently, so it is refused naming both kinds. GET /uploads/genotypes lists them, /{name} its metadata (overlap, new individuals, kind, digest), /{name}/markers pages the marker table, and /{name}/geno writes the file and returns the path for `genotype_file` on the compute endpoints. DELETE removes one. scripts/example-genotype-upload.py turns any .geno into a valid update -- a subset of markers, a few individuals dropped and added -- and tests/genoupload.rs builds its own from the deployment's BXD.4 for the same reason: upload, page the markers, write the .geno and check the layout, delete, then break each rule (no overlap, numbers in a letters panel, commas) and check the refusal says which. Unit tests cover the three value kinds, missing cells, ragged rows, a non-numeric position, the 50-new-individual limit, and the .geno layout (Chr first, then the marker -- GN's order, not the upload's).
8 daysmatrices: upload new traits and covariates as one named tablePjotr
The other half of changing data: bringing new columns in. A matrix is a delimited table -- a header row, then one row per individual, the first column naming them -- stored in the uploader's own SQLite file and checked against an existing genotype version's panel before it is accepted. * POST /matrices {name, genotype_version, contents, kinds?}: parse the table (tab, comma or semicolon, guessed from the header), then check it against that version's individuals. The count must match and every name must be in the panel; 92 of 93, or a name the panel does not have, is a 422 that says which. A mapping that silently drops individuals is worse than an error. * kinds marks columns that are covariates; everything else is a trait. * Each column becomes a basket named <matrix>.<column> -- a dot, not a slash, because a basket name has to survive a URL path segment -- so a column maps on any engine with {"basket": "my-traits.bw"} and inherits the revision history and digest without a line of new mapping code. * GET /matrices lists them, GET /matrices/{name} their columns and basket names, GET /matrices/{name}/values the table back (parsed values, null where missing), DELETE removes the matrix and its columns. Two bugs this found in the basket store, both real: * SQLite has foreign keys OFF by default, so the ON DELETE CASCADE never fired: deleting a basket left its rows behind, rowids were reused, and the next create hit "UNIQUE constraint failed: basket_version.basket_id, revision". The pragma is now set, and deletion removes the dependants explicitly, because files written before it have orphans. * The first attempt named a column's basket "<matrix>/<column>", which the basket-name rule (rightly) refuses. The dot keeps both rules. scripts/example-matrix.py writes a table from a genotype file's own header with deterministic values, so the fixture passes the panel check by construction -- tests/matrices.rs builds its own for the same reason and walks the loop: upload, read back, map a column with R/qtl2, delete, plus the three refusals (too few individuals, a name not in the panel, an unknown version).
8 daysbaskets: every revision kept, files named for the user, all four enginesPjotr
Three things the first cut was missing, and both were asked for. * An automatic version history. Revision 1 is the column as first stored and each PATCH adds one -- with its own digest and Unix timestamp -- and earlier revisions are kept. GET /baskets/{name}/revisions is the history, ?revision=N reads an old column, and a mapping can pin one: {"basket": "wbc-edit@2"}. That is the point of a digest: a result quoted yesterday can still be reproduced after the column has moved on. * Files named for the user, not a hash of the subject: BASKET_DIR/<user>.sqlite (0600, WAL). Tokens rotate, subjects do not, so a new token finds the same baskets -- and an operator can see whose baskets a file holds. A subject that is not filename-safe is sanitised and disambiguated with a short digest, and a pre-naming hash file is renamed on first use rather than orphaned. * /compute/rqtl and /compute/gemma and /compute/fitqtl take `basket` too, so all four engines share one meaning: a basket resolves to the same (name, value) pairs a client would have posted, and every reply records the revision and digest it ran. The schema upgrade is a real one, not a reset: an older file's single column becomes revision 1 of the history, cells copied across with the digest and timestamp they already had. Tested by building the old schema by hand and opening it (an_older_file_is_upgraded_with_its_column_intact), which is also how I found that the migration was recreating a table the base schema had already made. Verified on all four engines, head and pinned: rqtl 3811 rows, gemma 3809, qtl2 3730, fitqtl 3 markers -- each reply carrying "Basket '...' revision N (digest ...)".
8 daysbaskets: a user's own column of values, in their own SQLite filePjotr
The next phase is changing data, and the constraint is that MariaDB is never written to. A basket is the storage half of that: take a trait's values, edit some of them (or build a covariate column that does not exist), keep the column under a name, and map that instead. * POST /baskets stores a complete column; GET /baskets lists yours; GET /baskets/{name} its metadata; GET /baskets/{name}/values the cells; PATCH /baskets/{name} replaces the column with a new revision; DELETE removes it. * POST /compute/qtl2 takes "basket": "<name>" in place of `samples`, and both at once is a 422 rather than a silent choice. The mapping code below it is untouched -- a basket resolves to the same (name, value) pairs a client would have posted. * One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600, WAL, busy_timeout. `GET /baskets` only ever lists the caller's own; the file name is a hash, so a directory listing does not name users. * The full column is stored, not a diff -- the caller has the original beside it, and a mapping needs one complete column. * Text with a parsed copy beside it: raw (what was typed), value (the number or NULL), status (ok/missing/bad). "NA" and 0 stay different claims, and a column with something that is neither a number nor a missing marker (NA, x, -, ., ?) is refused with the samples named -- silently dropping it would change a trait without saying so. * A revision and a digest: an edit bumps the revision and changes the digest, and a mapping reply records both ("Basket 'wbc-edit' revision 2 (digest 9559a70e...)"), so two runs on one name cannot look alike. A result cache must key on the digest, not the name. The loop, end to end on BXD trait 10001: 34 values posted as a basket, one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2 -- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the revision and digest. tests/baskets.rs walks that loop (create, list, values, patch, map by name, delete) and the refusals: a non-number named in the error, a name that cannot become a path, an unknown basket, and basket-plus-samples. Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest changing on edit, per-user isolation, and name validation. rqtl/gemma take the same 10-line change next -- their `samples` shapes differ, so they are a separate commit rather than a rushed one.
8 daysapi: scans point at the class counts behind their own peakPjotr
The gotcha notes in SKILL.md tell an agent to count the animals per genotype class before believing a peak; a note in a document is only found by whoever reads it, so the replies now say it too, at the peak they just computed. * suggest::class_hint builds /groups/{g}/genotypes/{v}/markers/{peak}/classes?trait={trait_id} -- a real GET with one placeholder, because a scan knows the trait's values, not its PublishXRef.Id. * /compute/{rqtl,gemma,qtl2,fitqtl} add it to `next` after the PheWAS link, naming the peak each reply found and the version the request named (a {version} placeholder when it named none). * The genotype subtree carries the template too: a version's detail, and a concrete marker's own links. * /compute/qtl2 gains the sentence that explains the divergence: on a riself (riset) panel heterozygous calls are dropped before the scan (R/qtl's convert2riself) and the HMM re-imputes them, while GEMMA keeps them at dosage 0.5, so a mostly-H marker cannot peak there and two engines can name different markers of one region without either being wrong. tests/links.rs: a scan's reply offers the class counts at its own peak -- the link names the version and the peak marker the reply reports, and leaves {trait_id} for the caller.
8 daysgenotypes: a trait summarised per genotype class at one markerPjotr
The question that follows "why do GEMMA and R/qtl2 peak at different markers": what does the trait look like within each genotype class at one marker. The frequencies endpoints report the distribution of calls, not the values behind them, and the answer must not be "serve the matrix" -- this API never does. GET /groups/{group}/genotypes/{version}/markers/{marker}/classes?trait=<id> -> {"A": {"n", "mean", "sd", "min", "max"}, "H": {...}, "B": {...}} The calls are joined to the trait's per-strain values inside the server (the same lookup, authorization and 404s as /groups/{g}/traits/{id}/values) and reduced to those five numbers per class. Nothing in the payload names an animal. A class with fewer than ten animals is called out in notes, because a peak resting on two is a different claim from one resting on a hundred. It settles the DOL trait 10003 question. At GEMMA's top marker JAX00418132 (chr16 36.35 Mb) the classes are A n=2 mean 1.8, H n=20 mean 2.53, B n=171 mean 3.95 -- the A class is two animals and they are the trait's two lowest values. R/qtl2 drops H on a riself panel (R/qtl's convert2riself), so it keeps 173 animals there but only 2 of one class: nearly monomorphic, which is why its scan peaks elsewhere. GEMMA keeps the 20 hets at dosage 0.5, and that is where its sharper p-value comes from. GEMMA's next two hits are the same pattern in opposite directions (one A among 171, then two B among 171), so the chr16 block is a haplotype carried by a handful of extreme animals, not a population effect. Also fixes the reading of GEMMA's `af`: it is mean(dosage)/2 under the BIMBAM coding this deployment ships (B=1, H=0.5, A=0), so 0.469 for JAX00418132 is half the B-allele frequency 0.935, not the A-allele frequency 0.065. The bimbam file and DOL.geno agree at that marker. tests/frequencies.rs gains a route test: the class fields, A < B, the small-class note, the "not served" note, no individual names anywhere in the payload, and a 404 for a trait the group does not have.
8 daysgenes: Wikidata aliases, cached in gene-aliases.lmdb for seven daysPjotr
A GeneNetwork trait page shows the gene's aliases next to the symbol, and GN4 had no way to ask for them: gn-guile has a Wikidata lookup (gn/db/sources/wikidata.scm) and nothing in GN4 did. This is that lookup with the caching Wikidata deserves -- it is a shared public service and an alias list is not news. * GET /genes/{symbol}/aliases: two SPARQL queries against WIKIDATA_ENDPOINT (the entity: a gene wdt:P31 wd:Q7187 in human, mouse or rat, labelled with the symbol; then its skos:altLabel s in English), cached in WIKIDATA_CACHE_DIR/gene-aliases.lmdb, an LMDB file (NO_SUB_DIR, 0600, so one file and LMDB's -lock companion rather than a directory of two). * TTL WIKIDATA_TTL_SECONDS, seven days by default. Past it the value is refetched; ?refresh=true forces that. * The label is matched case-insensitively, which the gn-guile query did not: human genes are labelled SHH and mouse ones Shh, so an exact match finds only one species. ?species=human now returns the ortholog's labels -- HLP3, HPE3 and the rest -- which are the ones a mouse trait page shows. With no species named the pick is deterministic (mouse, rat, human), so a repeated request cannot change entity. * A failed refresh keeps the last value: the answer says stale: true, the envelope is partial, and a note explains it. Nothing cached at all is a 502 that says so. An empty alias list and an unreachable upstream are different facts, and only one of them is about the gene. Seven unit tests (key scoping, TTL including a clock that went backwards, the query shape and quote-escaping, SPARQL flattening, an LMDB round-trip that asserts the file is gene-aliases.lmdb, and stale-on-failure with and without a cache) and two route tests (the second call is served from the cache -- same fetched_at -- and the species switch), which skip with a 502 assertion when there is no network.
8 daysprobesets: cross-dataset lookups by probeset id and by gene symbolPjotr
A probeset id is not a Xapian field, so /search cannot find 1427571_at exactly and the only route to it was guessing a dataset and calling /datasets/{ds}/probesets/{name} until one answered. A gene symbol could be searched, but the reply was capped at 100 hits, interleaved probesets and platforms, and carried no total -- so "did I see all of Shh?" had no answer. Both questions are one indexed query each: ProbeSet.Name_IDX and symbol_IDX. * GET /probesets/{name}: one row per dataset that carries the probeset, with the panel (group, species, platform) and the scan's statistics (mean, LRS, LOD, locus, additive). Suffixed chip variants match too -- 1427571_at also finds 1427571_at_A -- and each row's `name` says which matched, so the answer never hides a substitution. LIKE's wildcards are escaped, or every Affymetrix `_at` would match any character. 155 rows over 155 datasets in 32 ms. * GET /genes/{symbol}/traits: the same rows filtered on Symbol, with a real total, ?sort=lod, ?min_lod=, ?group=, ?species=, ?platform= and paging -- "BXD datasets where Shh is a strong eQTL" is one call. 1225 rows over 17 datasets and 13 probesets in 38 ms. * Both serve metadata, not values. A token is enough for the index; trait values stay per-dataset and authorized per dataset, and `next` goes to /datasets/{ds}/probesets/{name}/values, which checks. There is no cross-dataset values endpoint, deliberately: that is bulk work and /groups/{g}/traits is the bulk call. The one-statement join took 5.2 s for Shh -- MySQL rescans InbredSet/Species per row -- so the 933 ProbeSetFreeze rows and the 102 panels are fetched separately and joined in memory: 70 ms each. nsamples is left out: it is a ProbeSetData count per row (+0.5 s) and the dataset record already reports strains.
8 daysapi: every response suggests where to go nextPjotr
A caller that knows only /version should be able to reach anything by following links. Until now `next` was thin and uneven: the genotype subtree offered none at all, the group record offered one, /version offered nothing, and two links it did offer could not be followed. * suggest:: now builds the graph: root_links (groups, species, mapping-methods, openapi), genotype_version_links (frequencies, marker names, a marker's location and tally, the group, the scanning actions), unlisted_genotype_links, marker_links (location, tally, marker list, panel tally, PheWAS at that marker), group_links and trait_links. Envelope::with_links / with_links_except add them, deduplicating and never suggesting the response the caller is already reading. * Wired through the genotype surface (listing, version detail, markers, location), both frequencies routes, the compute endpoints, groups, traits, both value routes, species, datasets, publications, accessions, search, mapping methods, phewas, correlation and /version. * Fixed what was not followable, because `next` promises a GET that returns 200 and a client is the one that finds out otherwise: - /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are gone; comput endpoints now link ?locus=<their strongest marker> (suggest::peak_marker_of, computed before the rows move). - /datasets and /publications are not routes; root_links offers the listings that exist (/groups, /species, /mapping-methods) and /openapi.json. - /search?q= with an empty query is a 400, so it is not a link. - the correlation endpoint's "/compute/correlation" was a POST in `next`; it is in `actions` now. - marker_links pointed at /markers/{marker}, which is not a route; it points at /markers/{marker}/location. tests/links.rs walks the graph two hops from a dozen seeds and fails on any link that does not answer 200, plus a check that a tally leads to the marker list and to a marker's location, and a location to that marker's tally and its PheWAS.
8 daysgenotypes: call frequencies, marker names, and a location per markerPjotr
Two questions the genotype surface could not answer: what a panel is actually called as, and which markers it holds. The listing gives counts and the compute endpoints give scans; a panel whose calls are not the codes its header declares, or a marker that is 99% missing, stayed invisible until a scan came back flat -- and then the search for the cause started at the wrong end. * GET /groups/{group}/genotypes/{version}/frequencies counts the calls in the version's .geno and returns them as call -> frequency ({"B": 0.4889, "D": 0.492109, "H": 0.018991}). Frequencies are over the calls that are present, so they sum to 1; missing codes (U, N, NA, an empty cell) are counted separately and reported in notes, because the missing rate is the diagnosis. * A probability-valued file is binned 0.05 wide (twenty bins over [0, 1], keys are the bin edges), since 0.4187 and 0.4191 are the same statement about a marker; ?bin= sets another width. A file with a few distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported exactly -- binning three values hides what is being looked at. * .../markers/{marker}/frequencies is one locus, and ?marker= on the panel route is the same call. Counting goes through an interner, so cost is one hash lookup per cell and one allocation per distinct call; a .geno over 256 MB is refused (413 genotype_too_large) rather than read, which is what HSNIH-Palmer_true (3.7 GB, 134927 markers x 6147 individuals) demanded. * The version's /markers now lists marker NAMES only, one column, and .../markers/{marker}/location returns {"name","chr","cM","Mb"} as named fields: a page of names stays small, and a position is a lookup rather than a table the client must hold. The same size guard applies, so an oversized panel's listing is a 413 in 3 ms. * Files no group lists -- a panel in preparation, a fixture, the DO's do/ cross -- live at /groups/unlisted/genotypes/{name}/... with the same four calls. `unlisted` is a namespace, not a group: only its genotype subtree answers. One prefix for the whole genotype surface beats a second spelling for the same tally, so the earlier /genotypes/{name} spelling is gone. * genotypes::resolve_named (moved out of the frequencies module, since it is genotype metadata) resolves a version, a .geno name, or a path relative to the genotype files directory; MAX_SCAN_BYTES is shared by every whole-file read. Tests limited to the feature: genotypes 3 (names-only columns, paging, location fields, marker_not_found, unlisted parity, 413), frequencies 5, frequencies unit 7.
8 dayscompute/qtl2: multiparent crosses, by namePjotr
A multiparent population cannot come from a .geno. That file carries two codes per marker (B/D, A/H/B) -- a collapse, not a founder allele -- and the collapse is not invertible, so DOL's 194 DO animals cannot be fitted as a DO however much reference data appears. What can be is a cross that ships its own qtl2 control file with founder genotypes, so the endpoint now takes one by name: {"rqtl2_cross": "do", "trait_name": "DO::10001 (WBC)", "samples": ...} * resolve_control_file looks in genotype/rqtl2/{name}.json, then {name}/{name}.json (do/do.json), then {name}.json under GENOTYPE_FILES; prepare_control_cross reads the cross's pheno, maps and founder_geno headers without converting anything. It reports crosstype, founders and whether the cross is multiparent, and refuses a multiparent crosstype with no founder_geno rather than silently scanning it as biallelic. * Samples are matched to the cross's own population by name and the strays are counted in notes; rqtl2_cross cannot be combined with group or genotype_file. * The gate is given individuals x founders, because a multiparent scan fits one coefficient per founder allele at every marker. * Autosomes only now applies to shipped crosses as well: the DO's X is left out (369 markers) instead of being fitted without a sex covariate, and the script subsets with cross[, chr = autosomal]. Measured on the DO (WBC, 742 individuals, 8 founders, 6044 autosomal markers): probabilities 174 s / 1.39 GB (cached), LOCO kinship 15 s (cached), scan 21 s. scan1 runs on one core for a multiparent cross because its forked path stalls on allele probabilities here -- 25+ minutes at 8 cores with no CPU consumed, against 21 s serial. Two bugs found by running an actual published trait through it: * do_pmap.csv is in Mb, not the bp qtl2 documents, so every DO peak was reported at 0.0 Mb. The map's unit is now detected from its largest position and printed. * The phenotype file was read with the default whitespace separator, so a label with a space ("DO::10001 (WBC)") split into three fields and read.table stopped with "line 1 did not have 3 elements". Both the pheno and covariate reads are tab-separated now. The R error extraction also joins R's wrapped continuation line, so a complaint reads as a sentence rather than "Error in scan(...) :".
9 dayscompute: R/qtl2 as a third scan, POST /compute/qtl2Pjotr
R/qtl sits between the existing two scans: Haley-Knott on marker genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with mean-imputed genotypes. Neither can analyse a multiparent population, and neither models relatedness from the actual genotype data. R/qtl2 does both -- HMM genotype probabilities, then an LMM with leave-one-chromosome-out kinship -- and it is what names the third peak on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24, against GEMMA's -log10 p 7.15 there). * src/qtl2.rs converts this deployment's own .geno into a qtl2 cross under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with the genotype-code map, the orientation and the paths written out explicitly. The shipped genotype/rqtl2/ control files cannot be used -- they point outside the data directory, declare no code map (every genotype reads as missing) and no orientation (individuals read as markers) -- and each of those produces a plausible flat scan instead of an error, which is why the conversion is ours. * Heterozygous calls are dropped on a riself panel, as R/qtl's convert2riself does. Left in, qtl2 reads the cross as multiparent and its scan1 takes a path the panel cannot satisfy ("Error in scan1_pg ... : No individuals in common"). * Autosomes only: X needs a sex covariate to be fitted properly and Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is restricted the same way. The note says how many markers were left out. * scripts/qtl2_scan.R caches genotype probabilities and kinship by a hash of the control file *and its directory*; two conversions of the same panel can have identical control files and identical file sizes, and reusing the first one's probabilities silently scans the wrong cross. It also takes the individual names from whichever shape the cross's pheno slot has -- a single-column pheno file leaves it a plain matrix, and the resulting empty phenotype reads as scan1's "No individuals in common". * Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply unchanged -- post the same samples as /compute/rqtl. * tests/qtl2.rs: the fixture scans and its notes name the cross; BXD trait 10001 peaks on chr8 under both scans; refusals carry missing_genotype / invalid_kinship. cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like R/qtl; a DO cross costs minutes of genotype probabilities, paid once per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
9 daysfix: DOL maps again on both engines, and the four stale testsPjotr
DOL (200 individuals, 112 913 markers, one genotype version) failed on R/qtl with "There are missing chromosome IDs" and on GEMMA with a bare wrapper bail-out. Two unrelated causes, both in our own code. **R/qtl: a genotype code collided with a chromosome name.** DOL's header says `@unk:X` -- X is the code for a missing genotype -- and the wrapper passed that to `read.csv(na.strings=...)` for the *whole* file, so every marker on the X chromosome lost its chromosome ID and `read.cross.csv` refused the map. The unknown code now applies to the genotype columns only. DOL scans: 112 913 markers in 42 s, where before it never got past reading the file. Same fix in scripts/qtl_fitqtl.R, which shares the read. **GEMMA: its leave-one-chromosome-out run cannot fit the Y.** With LOCO the DOL panel is scanned per chromosome, and the Y run dies inside BLAS -- `Enforce failed for N>0 in src/fastblas.cpp ... fast_cblas_dgemm` -- taking the whole request with it (a 502 whose detail is the wrapper's banner). The mitochondrial chromosome has the same shape of problem. `prepare` for the GEMMA engine now writes the bimbam pair without the Y and M markers (cached beside the individual subset, line-aligned so the geno and snps stay in step), and the note says what was left out: "46 Y/M markers left out (GEMMA's LOCO cannot fit them)". DOL maps: 106 515 rows in 49 s. **And the four tests that had been failing since before this work**, all stale expectations rather than product bugs: * `benchmark_questions` pinned two traits to "page 2" of a 100-per-page search when they rank ~190th and ~219th of 246 -- one re-index pushes them over. It now walks up to three pages and asserts presence and LOD, not the page; and the helper `search()` pages, because a benchmark row can name a trait that ranks outside the first hundred of its own query. * `no_db_mode_degrades_gracefully` asserted exactly one hit for "alcohol" from the index alone. The index grew; the behaviour under test is that the search still answers, within the limit, with the database-backed fields null. That is what it asserts now. * `search_proxy_scenarios` and `search_parity_replay` never pinned the engine they test, so an ambient `SEARCH_ENGINE=native` made the proxy scenarios answer from the local index: a 200 where a 422 was expected, and "expected 2 upstream calls, got 0" for every replayed query. Both now set `SEARCH_ENGINE=forward`. * src/genotypes.rs: `DEGENERATE_CHROMOSOMES`, `drop_degenerate_chromosomes`, `Prepared.markers_dropped`. * src/routes/gemma.rs: also the 500 that `eb5320d` introduced -- counting parents/F1s from `panel - keep` instead of subtracting the exclusion *set*, whose length can exceed the panel (subtract overflow). * scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: na.strings scoped to the genotype columns. * tests/{benchmark,no_db,search,search_parity}.rs: as above. Verified through the API on DOL trait 10003: R/qtl 112 913 rows, GEMMA 106 515 rows. gemma/rqtl/fitqtl suites green (14, 10, 4).
9 daysgemma: leave out individuals with no genotypes, which make the kinship singularPjotr
Asking whether the singular matrices could be fixed turned out to have two answers, one of them a real bug on our side. The bug: a panel can name individuals whose genotype row is empty, and an empty row makes the kinship matrix singular. GEMMA then reports "eigenvalues close to zero" and dies in LAPACK with `LUDecomp ... input domain error` -- a message about linear algebra for a problem that is a missing genotype. Measured on this deployment's own files: BXD-AE.4 136 of 344 individuals have no genotypes MDP 93 of 178 BXD-Heart-Metals.4 200 of 481 HET3-ITP.3 2867 of 2867 (the whole file is empty) Dropping them makes those panels map: BXD-AE.4's GWA step goes from a failure to 3.2 s, and MDP's kinship step from a failure to success. The individuals contribute nothing to a scan either way, so nothing is lost; `notes` counts them, and the "Genotype:" line no longer blames parents/F1s for them (`prepared.excluded` is "panel minus kept", which now includes both, so the two counts are separated). Not the cause, but worth naming: `AXB` fails earlier still, on its annotation -- `ERROR: Enforce failed for D1Mit296 in src/gemma_io.cpp ... ReadFile_anno` -- which is a broken `.snps` file, not a matrix. That wants a pre-flight check of the annotation (report the offending marker) rather than a GEMMA failure, and is not in this commit. Considered and rejected: a ridge on K (`K + epsilon*I`) would work mathematically but GEMMA has no option for it and the wrapper caches K by content hash, so it would mean patching the wrapper's cached matrix; `-gk 2` and dropping markers do not add rank; and the non-LOCO path is the one that does not run at all here. * src/genotypes.rs: `individuals_with_genotypes`, one pass over the bimbam file marking the individuals with any genotype at all. * src/routes/gemma.rs: filter `keep` by it before alignment (so the trait values drop with them), and say so in `notes`. * tests/gemma.rs: `individuals_with_no_genotypes_are_left_out` -- BXD-AE.4 maps, the note names the drop and the reason, and the genotype line no longer claims 344 individuals. * README.md: the behaviour and why it exists, under /compute/gemma. 307 tests pass; the same four as before do not.