about summary refs log tree commit diff
AgeCommit message (Collapse)Author
32 hourscorrelation matrix: clustered heatmap and its plot HEAD mainPjotr Prins
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix reply and `POST /plot/heatmap` draws it: the r matrix in that order, the dendrogram that produced it, and a diverging scale with blue negative and red positive. A reader can see *why* two traits are neighbours instead of trusting an ordering. Clustering decisions, all recorded because an unrepeatable clustering is worse than none: * distance `1 - r` -- the usual choice for correlations, and it keeps anti-correlated traits apart, which is what a heatmap of correlations should show; * **average linkage (UPGMA)** -- single linkage chains along a gradient and complete linkage makes one flat blob out of noisy traits; average is the compromise GN1's R heatmaps used too; * **deterministic ties and leaf order** (lower index first), so two runs on one matrix produce the same picture and plots stay diffable; * a matrix with a **gap is refused** rather than clustered: an unknown distance treated as 1 would invent a structure, and the reply's note says to drop those traits and ask again. Pairing with the PCA work this is the visual half of a collection: scree plot for how much structure there is, clustered heatmap for where it is. Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back. Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
33 hoursplot: the scree plot of a matrix PCA, which GN2 draws client-sidePjotr Prins
GN2 renders a scree plot with Plotly in the browser, from the explained variance ratios its matrix page computed. This is the same figure as a file: `POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns image/svg+xml, like every other plot endpoint -- the plot is of the numbers that were computed, nothing is recomputed. scripts/plot-pca.py draws a bar per component (its share of the variance, GN2's number), a cumulative line with the 80%-of-variance crossing annotated (what a reader actually uses to choose how many components to keep), the eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps in the matrix -- the one thing GN2 hides by mutating its shared-sample list. `kind=N` on the request caps how many components are drawn, since a 200-trait matrix would otherwise draw 200 bars. Also: the monitor's SVG judge now understands a binary reply (the client hands back content type and size, not the bytes), and `plot-pca` is a check of its own: it asks the matrix for a PCA and posts the reply to the plotter. Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative path, which only resolves when the service's working directory is the checkout). plot-pca and plot-genotypes both ok.
33 hourscorrelation matrix: PCA, behind its parity testPjotr Prins
`"pca": true` on the matrix request now folds the correlation matrix into principal components: eigenvalues descending, explained variance ratios, loadings per kept trait, and the eigenvalues' sum reported so a caller can check the decomposition is sane instead of trusting it. The eigendecomposition is cyclic Jacobi -- symmetric matrices, no LAPACK, no new dependency -- capped at 300 traits because Jacobi is O(n^3) per sweep and this is for a collection, not a dataset. Two decisions from ai-memory #103, both visible in the reply: * **Traits with any null cell are dropped and named**, with the reason. A component from a matrix with holes is not defined; GN2 silently mutates its shared-sample list instead, and nobody can reproduce that component. * **The sign is fixed**: within a component the largest absolute loading is positive. Eigenvectors are defined up to sign, so without a rule two correct implementations disagree and no parity test could be exact. Verified against numpy.linalg.eigh on the same r the reply carries (scripts/pca-parity.py, BXDPublish 10001/10002/10003/10005/10007): eigenvalues: worst relative difference 3.2e-14 loadings: worst absolute difference 9.99e-16 trace: 5.000000000000 for 5 traits Unit tests cover the identity (unit eigenvalues), the closed form for 2x2 (1 +/- rho), the trace, a non-symmetric matrix being refused, and the cap. The monitor gained `compute-correlation-pca`, whose judge checks that the eigenvalues sum to the trait count and the ratios to 1.
38 hourscwl: make the ravanan path use the gemma build the wrapper path usesPjotr Prins
`(specifications->manifest (list "gemma"))` resolved to Guix's `gemma`, which is 0.98.5 (banner 2021-08-25); gemma-wrapper pins `gemma-gn2`, which is 0.98.6 (banner 2022-08-05). Same source, different build, and the two paths disagreed in beta by ~3e-3 because of it. Now both name `gemma-gn2`. Verified on the job side: the latest runs' own logs report GEMMA 0.98.6 and OpenBLAS 0.3.31, the same pair the wrapper profile resolves to. The remaining difference is smaller and has a different cause, measured rather than guessed: same path, two cold runs: worst |Δbeta| 5.1e-6 (1034/21055 bit-identical) wrapper vs ravanan: worst |Δbeta| 3.1e-3 their kinship matrices: differ by ~1e-5 cell-wise, same 198 individuals So GEMMA is not reproducible run to run under multithreaded BLAS (both contexts report 96 threads), and the kinship matrix amplifies that by roughly two orders of magnitude into beta. Pinning threads on both paths is what would make the parity gate exact; until then 0.005 is the honest tolerance, and scripts/gemma-compare.py now defaults to exact and says which one it is enforcing. Also notes in the comment that a manifest change alone does not take effect: ravanan keys a step on (script, inputs), so it kept reusing step scripts built against the old manifest until the workflow source changed.
38 hoursgemma-compare: make the parity gate run, and say what it is measuringPjotr Prins
The script could not have caught anything: it posted without an Authorization header (401 against a token-protected server), asked for a trait's values as `{"name": n, "values": [<trait name>]}` (422: samples need numbers), read `columns`/`rows` from the envelope's top level (they are under `meta` and `data`), and compared relatively -- which calls beta 0.0131 against 0.0157 a 16% disagreement on a marker where both runs agree to three decimals. Now: `--token`/`$GN4_TOKEN`, trait values fetched from the API, the envelope shape handled, and an ABSOLUTE `--tolerance` (default 0 = exact). It fetches, compares and reports through the server as deployed. Verdict on the current deployment, BXDPublish 10001 over 34 individuals: rows: gemma-wrapper 21055, ravanan+gemma 21055, shared 21055 worst absolute difference: rs230640887 beta -1.015599 vs -1.018739 (0.00314) --tolerance 0.005 -> agree; --tolerance 0 -> 63 163 values beyond it So the row difference this started from is gone: same markers, same count. The residual is that the two paths run *different GEMMA builds* -- the wrapper's log names gemma-gn2-0.98.5-8cd4cdb, the ravanan manifest gemma 0.98.6 -- which moves beta/se/p_lrt in the third decimal while leaving af, n_miss, l_mle and the marker set identical (and the top hits in the same order).
38 hourscwl: name the caching model, so nobody hunts a gemma-wrapper cache that is ↵Pjotr Prins
not there ravanan's store keys a step on (script, inputs) and there is no wrapper cache in this path. Recorded next to the 'No gemma-wrapper' rationale because the detour was expensive: a /gnu/store script that did mention --cache-dir /tmp/gn4-gemma-ravanan looked current only because /gnu/store's epoch mtimes defeat ls -t.
38 hoursoperator checklist: the ravanan build is a patched one, and whyPjotr Prins
Three local patches, the third of which is the reason a 20-chromosome scatter now runs 20-wide instead of 1-wide: {"job": ...} versus {"jobs": [...]}, which is slurmrestd's heterogeneous-job form. With the node declaring 30 CPUs (cold 24.7 s) and the submit fixed (cold 9.4 s), ravanan is within 3x of gemma-wrapper's 3.1 s for the same rows instead of 33x.
2 daysgemma: suggest BulkLMM rotations for a threshold, with the numbersPjotr
GEMMA's reply offered R/qtl as the way to get significance thresholds, which is the wrong advice for a relatedness-structured panel: haley-knott's permutation shuffles phenotypes without regard to the kinship, so its null is anti-conservative exactly where an LMM was chosen in the first place. BulkLMM is the same model family as this scan and permutes by rotating the data through the kinship decomposition, so its thresholds do respect relatedness -- and it is fast. The 200 reply now carries three actions in order: draw it (/plot/gemma), POST /compute/bulklmm with a ready-to-post body (group, trait_name, samples, nperm 500, seed 1), then POST /compute/rqtl relabelled as a non-LMM cross-check with a warning that its threshold is anti-conservative here. Measured on BXD trait 10001, so the text asserts nothing it has not seen: GEMMA's best marker is rsm10000005700 on chr8 (p_lrt 5.9e-07, -log10 p 6.23) and BulkLMM's best is the same marker (LOD 3.866, beta -0.0615) -- at the peak the two LMM families agree on the locus. Their thresholds are not interchangeable: with 500 rotations BulkLMM's 0.05 threshold is LOD 3.78 (the peak clears it, and a second seed gave 3.62), while R/qtl2's LOCO scan1perm on the same trait and permutation count gives 2.62, so a global kinship is the stricter null; and LOD 3.87 and -log10 p 6.23 are different scales for one marker, so a BulkLMM threshold must never be compared with a GEMMA p-value directly. The 422 permutations_not_supported text quotes the same measurements instead of prose, and the suggested body was run verbatim to check it: POST /compute/bulklmm with it returns 200, 21056 rows, and the threshold note "500 rotations (seed 1): alpha 0.01 LOD 4.10, alpha 0.05 LOD 3.62".
2 daysqtl2: permutations for the LOD thresholdsPjotr
POST /compute/qtl2 gains nperm, perm_seed and perm_strata, and the reply gains meta.permutation {n, seed, significant, suggestive, values} -- the genome-wide maximum of every permutation, so a histogram needs no second call. `data` stays the positional rows array on purpose: /plot/rqtl posts it straight back and the monitor counts its rows, so the null went beside the rows rather than around them (a generic Envelope::with_meta/Meta::extra, skip_serializing_if, so no other endpoint's response changes). Thresholds come from R/qtl2's scan1perm over the same genotype probabilities, kinship and covariates as the scan, with chr_lengths from the map -- without those it derives chromosome lengths from marker positions and the numbers stop being comparable with R/qtl's. The permutation CSV is written in R/qtl's two-column shape so one parser and one percentile routine serve both engines: PermutationResult::from_values is now shared with /compute/rqtl instead of duplicated. Semantics, each with the note that says it: * scan1perm shuffles phenotypes across individuals -- right for an outbred cross, only approximate for a selfing RIL panel, where qtl2perm is the correct package (not installed here, and not packaged); * perm_strata takes a case attribute name (sex, Epoch, ...) and permutes within it, because shuffling across sexes inflates the null; * fewer than 100 permutations gets a note that the 0.05 quantile is a quantile of too few values; more than 10 000 is refused, since each permutation is a whole genome scan; * a seed is reported always -- without one, two identical requests would give different thresholds. Cost, measured on BXD trait 10001 (34 phenotyped, 20 chromosomes, LOCO kinship, 4 cores): 20 permutations 0.6 s, 100 in 2.2 s, 500 in 14 s, 1000 in 24 s through the API; 0.022-0.03 s per permutation. Thresholds converge as expected (0.05: 4.331 at 100 and 200 permutations, 4.374 at 1000 on the raw probe) and are reproducible: the same seed gives the same numbers on 1 and 4 cores. Four things that only surfaced by running it, all commented where they bit: optparse keeps the dash (--perm-out is opt[["perm-out"]], so opt$perm_out was always NULL and the file landed somewhere the caller did not look); perm_strata must be a named character vector, not a factor; the LOD in the CSV is the LAST numeric field (R/qtl's shape puts the permutation index first, and a first-field parser reads that as the LOD); and percentiles interpolate at (n-1)p/100, so 100 maxima give 0.6733, not 0.67. bin/monitor-gn4 gains compute-qtl2-perm (nperm 10, judging meta.permutation) so a deploy that loses the R path shows up as a red check rather than a surprise.
2 daysmonitor: run the cached engines cold, not just warmPjotr Prins
A warm run answers from caches the previous binary filled, so it can pass while the cold path -- what a new caller's first request actually does -- is broken. `--cold` sends each cached engine a payload with one sample value nudged by a thousandth, then repeats the original: every engine here keys its cache on content (gemma-wrapper's hash, ravanan's interned inputs, rqtl/qtl2/bulklmm's own), so the first is a miss and the second a hit. Both runs are judged; the line reports "cold X / warm Y". Measured on production with the 30-sample fixture: compute-ravanan 102.0 s cold / 3.7 s warm compute-gemma 3.1 s cold / 2.0 s warm compute-rqtl 7.5 s cold / 7.3 s warm compute-qtl2 3.3 s cold / 3.3 s warm compute-fitqtl 6.7 s cold / 6.9 s warm compute-bulklmm 19.6 s cold / 19.0 s warm ravanan is the one where the cache is the difference between minutes and seconds; the rest are compute-bound at this size, and the honest thing is to print the pair rather than assert a speedup. A check whose payload cannot be perturbed skips with that reason instead of reporting two warm runs as cold and warm. The operator checklist gains the cold run as part of deploying, with both costs named: ravanan's cold run takes minutes, and every cold run adds cache entries.
2 dayscorrelation: the matrix endpoint, and the rule that a deploy ends with a ↵Pjotr Prins
monitor run `POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}` references -- a collection, as GN2's matrix form posts it, which spans datasets -- and answers an N x N matrix of r, p and n. The five decisions that are not obvious, each recorded where a reader will look for them: * **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to Spearman mid-row once a Pearson r passes 0.999, which the self-pair always does, so its matrix is asymmetric by construction. Here the method is chosen once and r[i][j] == r[j][i] exactly. * **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine calls, not N^2. * **The diagonal is by construction**: r = 1, p = null, n = the trait's own count, so half the traits cannot come back as 1.0000000000000002 with a NaN p. * **Per-cell n is its own matrix**, and a cell under `min_overlap` (default 6) is null in r and p while its n still says how many points there were. GN2 accepts 2, where |r| = 1 by definition. * **The sample axis is the first trait's group** (#106), resolved once and stated in the reply with the number of individuals it kept. Fetches are one query per dataset (`get_dataset_values_for_traits`, `get_publish_values_for_traits`), not one per trait: a matrix chooses a handful out of twenty thousand. Capped at 1 000 traits, because the cost is quadratic in the correlations and in the reply. No PCA -- the r matrix has to be trusted on its own first. Verified twice: cells agree with scipy to the last bit (BXDPublish 10001/10002/10005: 0.6756270426459695, 0.6950450737755965, 0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964, 0.8819879006700073), and a cell equals what the row endpoint says for the same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix, `min_overlap` above every cell's n, an unknown trait, `bicor`, and no traits at all. The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier all`, and says why: `/version` can report a commit and a reachable database while every engine that shells out is dead. The monitor gained a matrix check whose judge enforces square, symmetric and a unit diagonal.
2 daysmonitor: discover a second published trait, so the partial axis has a controlPjotr Prins
The check was written but never built: discovery kept one trait id, and the control comes from a second one. With this, --tier compute is 10 ok, 0 failed and 'compute-correlation-partial' answers 5 rows in 774 ms.
2 dayscorrelation: verify the partial axis against numpy/scipy, and clamp a perfect rPjotr Prins
The implementation had already landed in 698e6ab by accident -- a `git add -A` swept it in with the operator-checklist docs, which is why its message says nothing about partial correlation. This commit is the part that matters: it is now verified, and three things are fixed. **Verification.** The maths is least squares with an intercept, residuals, Pearson of the residuals, and Student's t on n - controls - 2. Checked two ways against independent code: * BXDPublish 10001 with 10002 held constant, target 17164, n=6: ours r=0.9960864631522379 p=0.00029371957431469716, numpy lstsq + scipy.stats.t.sf r=0.9960864631522379 p=0.00029372. Zero-order: ours -0.10577609106505677, scipy pearsonr -0.1057760910650567. * The same for 17084: identical to 16 digits, r=0.9840718340719924. A note on that first row: holding one control constant takes a target from r=-0.11 to r=+0.996 on six points. That is what partial correlation does when there is almost nothing left to correlate -- it is why the reply carries zero_order_r and zero_order_p beside it. **Fixes.** `t_two_sided_p` was wrong: the Lanczos `ln_gamma` had its `tmp` term sign-flipped, which still yields a smooth monotone function, so nothing crashed -- every p-value was simply false (a p of 1.8e-24 came out as 1.0). The corrected version matches `scipy.stats.t.sf` to 1.7e-12 relative over 64 (t, df) pairs and the df=2 closed form to 5e-14; both checkpoints are now tests. A perfect partial r (self-correlation) is clamped to +-1 with p=0.0 like the sample axis, and the reference name no longer repeats "against <dataset>" against the handler's own sentence. The monitor now exercises the axis: `compute-correlation-partial`.
2 daysmonitor: a check may decline to run, and the genotype plot follows the real flowPjotr Prins
Two separate faults behind one red line. The check posted `{group, markers: []}`, which is not this endpoint's request at all -- /plot/genotypes draws the numbers it is sent (`marker`, `genotypes`, `values`), it looks nothing up -- and the trait it picked (BXDPublish 10001, LOD 2.93) is *below* the fence that /peak-genotypes enforces (LOD >= 3), so even a correct payload would earn a 422. So: checks can now build their body from earlier replies (`body` may be a callable), and can decline with a reason (`Skip`) instead of going red -- calling a deliberately fenced endpoint's 422 a failure trains a reader to ignore red. The check now walks the documented chain: a published trait whose LRS is at least 14, its peak-genotypes calls, its values, then the plot. Full monitor after: 38 ok, 0 failed, 1 skipped, 0 busy.
2 daysmonitor: a --filter for "just retest this", and track the toolPjotr Prins
`--only` needs exact check names, which is no help when what you know is the dependency ("julia", "Rscript", "slurmrestd"). `--filter` matches a substring against the check's name, the sentence describing what it needs, and its hint -- so `--filter julia` finds compute-bulklmm even though its name says nothing about Julia. It selects tiers like `--only` does, and says so when nothing matches *because* discovery has not run (the compute checks do not exist until a group with samples is known). Also un-ignores bin/: the monitor is source, and an ops tool that is not versioned is the problem ai-memory #38 already named. Bytecode stays out. Co-report: the whole compute tier is green now (9 ok, 0 failed), including BulkLMM -- the recipe and the depot quirk are in the operator checklist.
2 daysoperator checklist: the tool block env.sh must keep, and why BulkLMM cannot ↵Pjotr Prins
work yet Writing down what the monitor found and what fixed it, because it is config that lives outside the repository and is therefore easy to lose at the next deploy: with only the cache dirs set, every engine that shells out answers 502 while the whole read surface and search look healthy. Measured on production after the fix (bin/monitor-gn4 --tier compute): 8 ok, 1 failed -- correlation, gemma (21 055 rows), rqtl, qtl2 (20 453), fitqtl, ravanan (21 055 rows), bnw, bnw-examples. Before: 1 ok, 7 failed, 1 busy. The remaining failure is BulkLMM and it is not a config line: the Julia project is registry-based and no depot on this box has its packages, so it stops at "Run Pkg.instantiate()". Populating a depot downloads from the Julia registry, which should be a decision rather than a side effect.
3 daysgitignore: ai-memory's sqlite journals and bin/ are runtime, not sourcePjotr Prins
They were tracked, so every deploy reported a dirty checkout and /version said "<commit>-dirty" -- which makes the one signal that says "this binary is the commit it claims to be" useless.
3 dayscorrelation: tissue and literature axes, paired the way the data actually isPjotr Prins
`type` on /compute/correlation now selects GN2's `corr_type`: sample (default), tissue or literature. Partial is refused by name rather than silently treated as sample. **tissue** reduces each gene symbol to its highest-Mean probe in TissueProbeSetXRef and pairs the two value vectors **by TissueID**. That last word is the difference from GN2, which keeps each vector as a bare list and pairs by position on the assumption that every probe covers the same tissues. On this database it does not: every probe carries exactly 26 values (liver, kidney, spleen, ... peripheral) or exactly 32 (striatum, hippocampus, neocortex, ... brain regions), the two sets are disjoint, and 45 281 probes are in each. Pairing those positionally correlates liver expression with striatum expression because both happen to be entry one. Pairing by tissue means a cross-panel pair has no correlation at all, which is the honest answer and makes the panel boundary visible instead of inventing a number. Verified independently for Shh vs Hcrtr1: 0.7222718302270719 from the endpoint, 0.7222719072696666 from a separate Pearson over the same TissueID-aligned pairs (the residue is MySQL's single-precision `float` column as printed by the client). **literature** looks the pair up in LCorrRamin3 in both directions after translating rat/human ids through GeneIDXRef, in chunks of 4 000 genes (the first attempt died on MySQL's 65 535-placeholder limit). It is a precomputed value, not a statistic: p and n are null. A trait with no gene id or no row is left out and counted in the note, where GN2 reports 0.0 and fills the tail of its top-N with them -- that reads as "measured, no relationship", which is not what happened. Gene ids come from ProbeSet.GeneId, not ProbeSetXRef.GeneId: the latter has no such column here, which GN3's own mapping also records.
3 daysdocs: what is missing from genenetwork4, ranked by migration effortPjotr
A survey of the three clones side by side (genenetwork1, genenetwork2, genenetwork3) for functionality that would fit a JSON API, ordered by what it would cost to migrate rather than by which repo it came from, in three tiers: S (hours to a day), M (days), L/XL (weeks or a new concept). Each row says which of GN1/GN2/GN3 carries the feature -- yes, part or no, established by reading their code -- so a row is not mistaken for something nobody has ever built. Nine rows are S, eight M, eight L/XL; the S tier is almost entirely metadata, which is exactly the pattern the size evidence explains: ProbeSetData 5 314 694 464 rows, LCorrRamin3 127 368 780, SnpAll 79 534 772 and GeneRIF 8 425 all in db_webqtl, so every value-shaped feature is a query-design problem while every metadata feature is small. Also lists what genenetwork4 already covers, the recommended order (matrix, then tissue and literature correlation, then pagination and the dropdown tree, then the sample-matrix and genotype exports, then partial correlation), and where each feature lives in each clone. ASCII only, aligned so the tables read as tables in a monospace view.
3 daysoperator checklist: how production actually runs now (store binary, no ↵Pjotr Prins
container) The layout, the restart command, where mutable state lives, and the two mistakes that cost time today: the CWL comes from the checkout rather than the store, and only two of the four ravanan 0.2.0 builds on this box convert --slurm-api-endpoint to a URI -- the others die inside slurm-api.scm with struct-vtable on a string, after staging every input, which reads like a Slurm fault rather than an argument one.
3 dayscorrelation: the live site answers Spearman whatever method it is asked forPjotr Prins
Recorded next to `Method` so the next comparison does not chase a 0.005 disagreement in r for an afternoon. GN2's `get_sample_r_and_p_values` takes a parameter called `type` and is called with `corr_params['type']` -- "sample" -- where the method belongs, so `if type == 'pearson'` is never true and `scipy.stats.spearmanr` always runs. The method is used only on the tissue path. Evidence, from the live site with `method=pearson&type=sample`: 1427571_at vs 1447043_at reported -0.8446202819617522 spearman -0.8446202819617523 pearson -0.8393085803732605 1427571_at vs 1427641_at reported -0.8479321530363810 spearman -0.8479321530363810 pearson -0.8451666639589470 BXDPublish 10001 vs 17084 reported 1.0 spearman 1.0000000000000000 pearson 0.9757237677298747 Asked for spearman, our endpoint reproduces the live site's whole reply: the same five traits in the same order, r to 2.2e-16, p to 5e-10 relative. Asked for pearson it reproduces the Pearson r of the values the same site publishes, to 1.4e-8 (their sample_data endpoint rounds to three decimals). The live site stays the reference for the sample axis, the six-pair overlap, the |r|-descending order and the reply shape. The statistic it is asked for is the one place where it is not: it ignores the request.
3 dayscorrelation: run over the group's sample axis, and report a perfect ↵Pjotr Prins
correlation as 0.0 Two divergences from the live genenetwork.org, both measured against it. **The sample axis.** GN2's `get_sample_r_and_p_values` walks `target_dataset.group.samplelist` and takes the reference value by name: the correlation runs over the group's animals, not over every individual the trait happens to have a value for. Trait 1427571_at of HC_M2_0606_P carries 99 values, the BXD panel holds 67 of those individuals, and the answer changes materially -- GN2's top hit is a -0.845 probe that does not appear in our top five at all. `corr_samples_group` now takes `samples_primary` (default, the group's genotype panel, which is where GN2's list comes from) or `samples_all` (the old behaviour); the panel is read with `genotypes::resolve`, falling back to the group's used-for-mapping strains where a group has no genotype file on this machine. `samples_other` is refused rather than guessed. **A perfect correlation.** The engine's r can land at 1.0000000000000002, which makes 1 - r^2 negative and its p-value NaN. The live API answers 0.0 for that row (scipy's pearsonr), so the reply now clamps: r to +-1 and p to 0.0. That is the reference's *output* being reproduced, not a computation being redone -- #96 stays resolved in the same direction. With the axis fixed, n agrees with the live site exactly (67) and r agrees with GN2's own published per-sample values to 1.4e-8. What does not yet agree is GN2's reported r itself: over the same 67 individuals, its `/correlation` endpoint says -0.8446202819617522 for 1447043_at while correlating the values its own `/sample_data` endpoint publishes for the same two traits gives -0.8393085803732605; on a second pair the gap is -0.8479321530363810 against -0.8451666639589470. Both pairs are off in the same direction and by a similar amount, so the two GN2 endpoints are reading different values for the same samples -- not a difference of sample sets (all four BXD genotype files give the same 67) and not a transform (2**v, 10**v, exp, v**2 and log were all tried). Until that is settled, our numbers are the reference's published values correlated correctly, and the ranking cannot be identical to a number we cannot reproduce from the reference's own data.
3 dayscorrelation: port the reference suites' cases, and note what is not portablePjotr Prins
The cases genenetwork2 and genenetwork3 assert, copied so GN4 answers the same questions the same way: - genenetwork3 test_normalize_values: only pairs where both sides have a value survive; - genenetwork3 test_generate_input: the dataset row keeps its positions, a missing value is an empty field rather than a skipped one; - genenetwork3 test_parse_results, with its own fixture file (data/correlation/sorted_results.txt): |r| descending, positives and negatives interleaved; - genenetwork3 test_get_samples: an unparseable value never reaches a correlation, and only shared sample keys are used; - the overlap boundary GN3 encodes as `num_overlap > 5`, which is where MIN_OVERLAP = 6 comes from. What is *not* portable is most of their suites, and that is worth writing down rather than discovering again: GN2's correlation unit tests are all mocks (compute_corr_coeff_p_value patched out, header strings, a three-value tuple), GN3's test_correlation is plumbing around the same mock, and test_rust_correlation tests the file/JSON layer that this implementation deliberately does not have -- it calls the engine in memory. None of them assert a single r or p, because those come from scipy at run time. One port needed widening: GN3's get_samples fixture has four shared samples, which is below its own minimum overlap, so asserting a correlation on it is impossible in both implementations; the fixture is widened and the assertion is that an unparseable value costs exactly its own pair.
3 dayscorrelation: phenotype traits and datasets, the common casePjotr Prins
Either side of a correlation may now be a Publish (phenotype) dataset, not just a ProbeSet one: a pheno trait correlated against an expression dataset is what GN2's mapping page is usually used for, and returning 404 for BXDPublish was the wrong answer. A publish reference trait is addressed by its numeric PublishXRef.Id -- that is how GN2 addresses phenotype traits, and a name there is refused with a message saying so rather than a confusing "no such trait". The bulk fetch gains the publish equivalent, keyed on the freeze's InbredSetId, so the same individuals line up on both sides. Measured live: BXDPublish/10001 (34 observations) against HC_M2_0606_P, top 5, HTTP 200 in 5.0 s; 1440079_at at r -0.757 (p 3.2e-6) leads.
3 dayscorrelation: the sample-axis endpoint, synchronousPjotr Prins
POST /compute/correlation now computes instead of returning canned data: one reference trait against every trait of a target dataset, observations being individuals, pearson or spearman, top-N by |r|. The reply is positional -- trait, r, p, n (column names in meta.columns) -- with the method, the minimum overlap and the top-N stated in the notes. `bicor` is refused with the methods that exist; a bad top_n is refused rather than clamped; a dataset or trait with nothing to correlate says so instead of returning an empty list. src/db.rs gains the bulk fetch the sample axis needs: every value of every trait of a dataset restricted to the reference trait's individuals, one record per (trait, individual, value), ordered by trait so the caller groups as it streams. It never loads a dataset whole -- ProbeSetData is 5.3e9 rows, so the query narrows by dataset and by individuals, which is also why the sweep, not the arithmetic, is the whole cost. Measured on the live db_webqtl: HC_M2_0606_P / 1427571_at against its own dataset, top 5, HTTP 200 in 16 s; the trait correlates with itself at 1.0 over 99 observations, then 0.802 (p 1.9e-23), 0.790, -0.789, -0.788. Synchronous is affordable at this size; GN2's job indirection exists for top-N 20000 across datasets, which is the next thing to time. Known and deliberate for now: the publish/genotype axes are not wired (the fetch is probeset-only), and a perfect correlation reports p as null -- GSL's t statistic is undefined at |r| = 1, whose r comes out as 1.0000000000000002, so 1 - r^2 is slightly negative and the engine returns NaN. Reproducing the engine exactly, rather than "fixing" the p-value here, is the decision recorded in docs/correlations.md.
3 dayscorrelation: the sample-axis engine path, with the decided semanticsPjotr Prins
src/correlation.rs now holds the part the engine does not offer in memory, and nothing of the arithmetic: `Pearson`/`Spearman` and their p-values are correlation_rust's (and GSL's). What is ours is the alignment into the engine's parser, the minimum-overlap gate, the ordering and top-N: - alignment is by individual, never by position: an individual the target trait lacks becomes "NA", which the engine's parser drops together with its reference value, so a missing observation costs one pair, not the row; - minimum overlap 6, so a two- or four-point +-1 cannot appear; - ordering reuses correlation_rust's own `custom_float_sorter` -- |r| descending, NaNs last -- so GN2, GN3 and GN4 sort identically; - top-N after the sort, clamped to GN2's 20000 ceiling; - bicor is refused by name with the methods that exist, rather than answered with something else. Tests cover both methods, the six-pair minimum, name alignment across a reordered and sparse candidate, the ordering with top-N, and the method parsing. The route is still the stub; wiring it to the bulk fetch is next.
3 dayscorrelation: vendor the engine's git dependencyPjotr Prins
Cargo.toml pins gn-rust-correlation 12c2a37 from the genenetwork forge (refs/heads/main there; GitHub still sits at 473ad82). The lock grows by exactly the four packages this brings in -- correlation_rust, GSL, GSL-sys, assert_approx_eq -- and nothing else moves. Following the xapian-rs precedent for a git dependency, since guix's cargo build is offline and the registry replacement only knows crates.io: - rust-package.scm gains the git origin as a bare `origin' (not a crate-source: unpack-rust-crates copies a *directory* input into guix-vendor/, so a checkout is exactly what it wants), plus crate sources for the three new registry crates; - guix.scm lists them and adds a second source-replacement stanza beside xapian's, keyed `git+<url>?rev=<rev>` because cargo resolves a git source by the exact id in Cargo.lock; - both git checkouts get the empty `.cargo-checksum.json` stub; - gsl is a native input: the p-values come from GSL's t-distribution through rgsl, found with pkg-config at build time. One fix in the vendoring phase itself: the gawk that back-fills package checksums named vendor directories from the lock entry verbatim, which is right for lowercase crates with no underscores and wrong for `GSL` and `GSL-sys` (crate-source lowercases, via crate-name->package-name). It lowercases now, so the two agree. Cargo cannot fetch this repo at all -- its HEAD is dangling, so cargo's "fetch HEAD and every head" fails where guix's fetch-by-sha does not. Generating the lock therefore needed CARGO_NET_OFFLINE with the crate pre-cloned into ~/.cargo/git/db; nothing about that affects the build, which vendors the checkout.
3 dayscorrelations: the design decisions, and the reference is reachable after allPjotr Prins
Synchronous, no redis, no job: one POST computes and answers. Use the correlation_rust engine in-process (already packaged, rev 473ad82), reuse its maths rather than rewriting the p-value, and implement only the sort and top-N. Minimum overlap 6 (gn3/python, not the engine's 4), pearson and spearman only, the engine's exact |r|-descending/NaN-last ordering, top-N as a request option applied after the sort, and the sample axis first. Verification is against the live genenetwork.org mapping page: it answers HTTP 200 in ~31 s, so the earlier "-m 20 curl" made it look unreachable when it is merely slow. Budget ~31 s per reference page in the parity loop.
3 daysdocs: what GeneNetwork correlates, and what correlation_rust doesPjotr
Answers the question that blocks duplicating the correlation_rust path: what is actually correlated. Four kinds over three axes -- sample (one trait vs every trait of a dataset, over individuals), tissue (gene vs gene over 26-32 tissue systems), literature (a table lookup in LCorrRamin3, no n at all) and partial (conditioned, GN3 only, async). Records the engine's contract as it is: JSON in, CSV out, one trait per line positionally aligned to x_vals, non-floats dropped pairwise, rows under 4 pairs refused, output sorted by |r| descending with NaNs last, top-N applied afterwards in Python, and no bicor in the engine at all -- which matters because GN3's Python path uses a minimum of 6 and the engine uses 4. Measured read-only against db_webqtl: ProbeSetData 5 314 694 464 rows, PublishData 3 288 638, TissueProbeSetData 90 562 DataIds with 26-32 values each (avg 29.0), LCorrRamin3 127 368 780. Hence no path loads a dataset whole, and the value fetch -- not the arithmetic -- is the cost. Ends with the two shapes a GN4 integration can take (thin: keep the file dance; direct: reuse parser::parse_rows_with_names and Pearson/Spearman in memory) and the four semantics that must be decided either way.
3 daysravanan-slurm: what happens when Slurm goes awayPjotr
Measured against the slurm container, 20 chromosomes, 40 jobs: baseline 200/21 055 rows in 205.5 s; slurmrestd unreachable -> 502 in 3.7 s; container killed 45 s into a request -> 502 in 50.2 s with 0 rows (no partial results); the same request after Slurm returns -> 200 in 163.3 s, i.e. the retry reuses the steps that had finished, because ravanan's store lives in RAVANAN_CACHE_DIR and not in Slurm. So: no automatic failover, no fallback to single-machine, nothing sticky, and a retry that is cheaper than cold. The 502's detail is ravanan's Guile backtrace, so the one useful line sits at its end; a pre-flight /slurm/v0.0.41/ping would make that a 503 naming the endpoint, and there is no timeout on the ravanan child for a Slurm that accepts connections and never runs the job -- both recorded as gaps.
3 dayscache: say when ravanan's cache cannot survive a restart, and document what doesPjotr Prins
Ravanan's reuse key is (script, inputs) and both live in its --store under RAVANAN_CACHE_DIR, so the cache's lifetime is that directory's, not the container's. Measured: 13.6 s cold, then 3.6 s for the identical request after restarting the container, with the store unchanged at 14 files -- Slurm's own spool state is lost and irrelevant to reuse. The default cache dir is under /tmp, which is neither durable nor visible inside a container that bind-mounts one host path, so a job cannot read the interned inputs. The reply now carries a note when the cache dir is under /tmp, naming the fix, rather than leaving it to be discovered as a slow or failing run. docs/ravanan-slurm.md gains a "What survives a restart" table (ravanan store, scratch, token registry, gemma-wrapper cache, Slurm state) and a path checklist; the README points at it.
3 daysgemma: read ccwl's YAML when checking the workflow's inputs, and fix two ↵Pjotr Prins
stale tests `ccwl compile` emits CWL as YAML (its only other target is dot), so the new input check must read that shape -- it was parsing JSON and would have turned a working request into a 502. A hand-written scan of the `inputs:` block, deliberately not a YAML dependency: ccwl's output is machine-generated and unvaried, and adding serde_yaml would mean regenerating the crate list in guix.scm for one small check. Two tests were stale in ways only the package's own check phase shows: - tests/auth.rs built AuthToken literals without the new `scheme`/`id` fields, so the crate's integration tests did not compile; - test/python/test_envelope.py asserted that /version carries no `next` links, which stopped being true when every read endpoint gained them ("always offer the next step"). The deployed build fails that test too, so it is not a regression from this work -- but it did block `guix build -f guix.scm` at the python-rest-test phase. Verified with the packaged binary (guix build -f guix.scm, all phases including both test suites): POST /compute/gemma/ravanan through ravanan and the Slurm container returned 200 with 636 rows in 13.6 s, on a request authenticated with a gn4ai_ short token -- the log line naming user zachs, scheme opaque and token smoke-2.
3 daysgemma: refuse a workflow from another commit, by namePjotr Prins
cwl/gemma/gemma-loco.scm is read from the deployment's checkout at request time (GEMMA_RAVANAN_WORKFLOW), not compiled into the binary, so a checkout and a binary from different commits can meet. When they do, cwltool answers "Type mismatch for input `kinship_pheno_file'; expected `File' but got `null'" -- which says nothing about the cause. Found by pointing the deployed f64a708 binary at the current workflow. The compiled CWL now has its declared inputs compared with what this build is about to send, before anything is staged for ravanan: a missing input or an undeclared one is refused with both lists named, plus the instruction to take the .scm and the binary from the same commit.
3 dayslog: name the person and the credential, and resolve handles from the registryPjotr Prins
The access log said who (`user`) but not with what: with short tokens a request can be made by a shared credential, so "which person" and "which credential" are now separate fields -- `scheme` ("jwt" | "opaque") and, for an opaque token, `token` (its registry handle, e.g. `agent-ravanan-3`). Never the token or its hash, and the path still carries no query string. The line also gains `ts` (RFC 3339 UTC), so a redirected stderr file is self-describing rather than relying on journald's prefix. Hand-rolled civil-from-days rather than a new crate: one format, and no Cargo.lock drift. scripts/gn4-log.py reads such a log and shows it as a table with the handle joined against tokens.sqlite: the subject as the registry has it now (flagging a handle whose subject has been repointed), and the token's status today -- [revoked], [expired], [unknown] -- which is how "an agent was working at 14:02" is told apart from "an agent was working at 14:02 with a credential revoked since". It reads the registry read-only, passes non-JSON lines through in table mode (the process log interleaves backtraces), and offers --json and --summary for jq and for counting. CallerSlot now carries the whole Caller (scheme, id, subject) rather than just the subject string.
3 daystokens: short opaque API tokens in a single tokens.sqlitePjotr Prins
A second credential format beside gn-auth's JWT, for agents and scripts: `gn4ai_` + 56 body characters + 2 checksum characters = 64, from an alphabet with no look-alikes, so a typo is refused before the registry is consulted. Opaque by design -- nothing is stored in the token; every fact lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode 0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row, takes effect on the next request, and a token in a log line reveals nothing; lose the registry and every short token stops working. Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and GET /tokens/who are behind a router layer that refuses any caller whose TCP peer is not 127.x.x.x/::1, and any request claiming to come through a proxy. A layer rather than a per-handler check, so nothing of a remote request is parsed before it is refused; there is no configuration switch, because minting credentials must never be reachable from the network. A short token reads public data only: every token-only endpoint, and `#[gnauth(secured)]` endpoints for public rows; a confidential dataset is refused with a 403 that names the JWT route, since gn-auth cannot validate an opaque token (the on-the-fly-bridge and registry-grant alternatives were both rejected for v1). scripts/gn4-token.py create|list|revoke|who drives the routes, so there is one implementation of the format and the revocation rules; the token is printed exactly once. Tests: the localhost guard over all four routes (remote peer, no peer, proxied header), and mint -> authenticate -> who -> revoke -> 401.
3 daysskills: ravanan is the default GEMMA runner, gemma-wrapper is the referencePjotr Prins
docs/www/SKILL.md: the workflow example now calls /compute/gemma/ravanan, and a new "Which GEMMA endpoint" section says when not to: Slurm down or submission failing, the user asks, a field ravanan refuses (covariates, normalization), or verifying results -- where the wrapper endpoint is the reference and scripts/gemma-compare.py is the check. docs/dev/SKILL.md: the same rule from the developer side, in Testing -- verify mapping changes by sending one body to both endpoints and never 'fix' a difference by switching endpoints, plus the reminder that the wrapper's cache only hits from 5eb53fa on, so a stale gemma-wrapper on PATH looks like a broken cache.
3 daysravanan: key the kinship step on the animal set, so K is shared across traitsPjotr Prins
gemma -gk reads the phenotype file only to learn which individuals are analysed -- the matrix comes from the genotypes -- so the workflow now gives the kinship step a constant-valued pheno file carrying just the trait's missing-data pattern, and gives gwa the real values. Verified byte-for-byte: -gk on a real trait and on its constant-valued twin produce identical .cXX.txt files, and trait 2's assoc checksums are identical whether K came from the constant pheno or the real one. Consequence, measured on the BXD fixture: a second trait over the same animals costs +1.0 MB of store and 7.5 s (20 GWA jobs) instead of +13 MB and 14.6 s (a second kinship scatter), because the 20 K matrices are identical inputs for both traits. Three keys now: workflow (the ccwl source), kinship (genotype + snps + lmm + the NA pattern) and trait (kinship key + the values), so the compiled CWL and the kinship pheno file are shared and only the trait's pheno file and inputs YAML are per-trait. This is also stricter than gemma-wrapper's own key, which ignores the phenotype entirely and would reuse a K computed for a different set of animals.
3 dayslogs: one access line per request, naming the callerPjotr
The proxy logs the caller through x-gn4-user, but the process log -- the one an operator tails while a job runs -- named nobody, so "who started this GEMMA run" meant going to the nginx log or guessing. genenetwork4 now writes one compact JSON line per request on stderr: {"event":"request","method":"GET","path":"/groups/BXD","status":200, "took_ms":10,"request_id":"…","user":"zachs"} Decisions in it: `user` is absent, not "-", when no token was verified (a dash is a value a reader can mistake for a user name); the path is logged without its query string, because a query can carry a credential and has no business in a log; the token value itself is never logged, only its subject; and the line carries the `request_id`, so it sits beside the existing `event: error` lines for 5xx and one grep joins a failure to the request that caused it. `GN4_ACCESS_LOG=false` turns it off where the proxy's log is the record of truth. Verified on a probe instance: with the test token the line says user "gn4-test-stub", with zachs's token user "zachs", and an unauthenticated 401 has no user field at all. docs/www/operator-checklist.md claimed "genenetwork4 does not log the token's sub" -- true until now, which is exactly the kind of sentence that rots; it is replaced by the line above, what it contains and what it deliberately omits. README's x-gn4-user paragraph gains the same, next to the identity it already described.
4 dayscwl: record the cold and warm LOCO timings, Slurm includedPjotr Prins
Full 20-chromosome LOCO mapping on the BXD fixture: 14.3 s cold and 1.6 s warm single-machine, 275.6 s cold and 1.6 s warm through the Slurm test container -- 40 jobs, 7320 rows, identical .assoc.txt checksums in every case. Slurm costs ~6.9 s per job here against ~0.36 s single-machine, which is container and scheduling overhead rather than gemma: the first request is cheaper locally, and only a repeated request (answered from ravanan's store) is fast either way.
4 daysdocs: always offer one sentence of what to do next, unless disabledPjotr
The API already computes every reply's options -- `next` are GETs that answer 200, `actions` are the POSTs to run next -- and a caller that reads them learned nothing extra from being told "you could now do X" in prose. What was missing was the instruction: an answer that stops at the result leaves the reader to work out the graph. docs/www/SKILL.md gains "Always offer the next step": end every answer with ONE sentence built from that reply's own next/actions -- after a compute run, a lookup, a plot, an upload, a basket edit. Prefer the action href over a hand-built call. A refusal is an answer too: its next or detail names the alternative (bulklmm for a GEMMA permutation, /compute/gemma for a covariate fit). Do not stack suggestions; one next plus at most one genuinely different alternative. "Skip the suggestions" / "just the result" turns it off for the session -- it is the sentence that is disabled, not the content. And never propose a step the API cannot do: if nothing in next/actions fits, say so. README states the same bargain where the next-link promise is described: the server's half is that the suggestion is always reachable, the assistant's half is that it is one sentence and it is a link the reply actually carries.
4 daysravanan: name the workflow inputs by content, so a repeat request is warmPjotr Prins
compute_ravanan wrote <pid>-<nanos>-pheno.txt, -ravanan-inputs.yaml and -gemma-loco.cwl, so ravanan's (script, inputs) work-reuse key was new on every call and a repeat of the identical request recomputed all 40 steps. Naming them by a hash of the request's inputs (genotype path and size, snps contents, trait values, lmm) makes an identical request hit the store instead. Measured on the BXD fixture, full 20-chromosome LOCO mapping, single-machine: 14.3 s cold, 1.6 s warm, identical .assoc.txt checksums. For comparison the gemma-wrapper stack costs 11.9 s cold and 12.1 s warm -- its own cache did not hit on a repeat here -- so the ravanan path is now the cheaper one for any repeated request and the slightly dearer one for a first request. The run log stays per-call: it is what a caller streams by run_id. The three shared files are written aside and renamed, so two concurrent identical requests cannot read a half-written file.
4 daysravanan+gemma: call gemma itself, not gemma-wrapperPjotr Prins
gemma-wrapper was doing two things ravanan now does or GEMMA already does: the parallel fan-out (ravanan's CWL scatter replaced it long ago, which is why every job passed --no-parallel) and LOCO orchestration -- gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and takes the annotation file it needs. What was left was a content-hash K cache and a --json record with results packed into .tar.xz archives. So cwl/gemma/gemma-loco.scm now runs two gemma invocations per chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix> -lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat, deterministic names. The kinship matrix travels from one step to the other as a CWL File output, and each job's .assoc.txt and .log.txt are the workflow's outputs -- so compute_ravanan reads files instead of parsing a wrapper JSON record and unpacking archives, and the pve note comes straight out of the jobs' logs. Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows -- 636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper also needed Ruby and pfff on PATH, and named its outputs with the extension doubled (...1.assoc.txt.assoc.txt). /compute/gemma keeps gemma-wrapper, deliberately: that is what makes the two endpoints comparable rather than two copies of one stack. scripts/gemma-compare.py runs the same body through both and reports row-level differences and timings. The gemma-wrapper content-hash K cache is the one thing given up; a repeat request recomputes (271 ms of kinship on this fixture).
4 daysgroups: the design in /groups/{group}, and a drift check against GN2's ParInfoPjotr
Two follow-ups to the parental-design endpoint. /groups/{group} now carries the headline in its notes -- "Parental design: C57BL/6J (maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes the parents and F1s by default" -- plus a next link to the full design. "Which cross is this?" is two strain names, and needing a second request to see them was the only reason the endpoint had to be called first. A group with no metadata says so in the same note, pointing at parents_and_f1s.json and at `individuals.exclude`, rather than leaving the reader to wonder why nothing was dropped. The metadata is the same fact in two places (GN2's ParInfo, our JSON) and it drifted once already, silently: the paternal/maternal labels were swapped against GN2's unpacking, and nothing noticed because the exclusion rule takes both strains regardless of role. scripts/check-parental-metadata.py parses ParInfo with ast (no gn2 import) and compares it group by group, reporting disagreements per role and treating groups the JSON knows but GN2 does not as informational. tests/parental_metadata.rs runs it while a GN2 checkout is present (GN2_CHECKOUT, or a sibling path) and skips once GN2 is retired -- at which point the JSON is the only copy and there is nothing to drift from. It reports 23 groups reproduced, 1 extra (BXD-Longevity).
4 daysgroups: an endpoint for a family's parental designPjotr
GET /groups/{group}/parents-and-f1s answers the question the mapping notes only implied. It reports the F1 strains, the maternal and paternal strain (from the deployment's parents_and_f1s.json, ported from GN2's ParInfo -- all 24 groups), the @type of the group's genotype file, the individuals each role names where the metadata names them (BXD-Longevity does, most do not), and -- the part a caller needs -- which members of the current panel the default rule drops and which the `F1` name heuristic catches. Two things that came out of building it and comparison with GN2: * the JSON's paternal/maternal labels were INVERTED against GN2's reading. GN2 unpacks ParInfo as `f1, f12, maternal, paternal`, so BXD's maternal strain is C57BL/6J; our file said paternal. No mapping was affected (non_strains takes both regardless of role) but the metadata was wrong, and anything using "maternal" later -- reciprocal F1 direction, the @mat/@pat note -- would have been backwards. The file is rewritten from ParInfo itself (ast-parsed, not retyped), 23 groups' roles filled in and the 24th (BXD-Longevity, which GN2 does not have) left as it was. * an RI set has two kinds of F1 name: the design's (B6D2F1) and the panel's own intercross individuals (C57BL/6JxBXD037F1). The endpoint reports excluded_from_panel and excluded_by_name separately and says which is doing the work, because "0 declared, 37 by name" is otherwise confusing. Empty F1 names (C57BL-6JxC57BL-6NJF2 has two) are dropped rather than becoming exclusion entries that match nothing. genotypes::non_strains is rebuilt on the same typed parser (DesignRole / ParentalDesign), so the rule and the endpoint cannot disagree. tests: a unit test for the roles, empty-name handling and the unlisted-group source, and an integration test asserting BXD's maternal/paternal orientation and the note.
4 daysindividuals: select who enters a mapping, by namePjotr
Parents and F1s are excluded by default (as in GeneNetwork2/3), with include_parents_f1 to keep them all. That boolean was not enough for what was asked -- "include these, drop that substrain" -- and the exclusion list itself is data (parents_and_f1s.json, seven groups) that no request can reach. F1 names also differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH, C57BL/6JxBXD037F1 for the intercross F1s), so pattern matching is the wrong tool. Every group-mapping endpoint -- /compute/gemma, /compute/gemma/ravanan, /compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm -- now takes "individuals": {"include": [...], "exclude": [...]} applied after the parent/F1 rule, exclude winning over include. An unknown name is a 422 invalid_individuals whose message lists the panel: a silently dropped animal changes a kinship matrix without anyone noticing, so the failure has to be loud. genotypes::select_panel is the one implementation, and it returns what it did (dropped parents/F1s, re-included by name, excluded by name) so the Genotype note can say it: "197 individuals, 38 parents/F1s dropped; 1 excluded by name". The note now also spells out the distinction that makes this option treacherous: the kinship uses *every* individual in the genotype file, the association uses only those with a trait value -- GEMMA reads no phenotypes for the kinship. So "include the F1s" changes the relatedness structure without putting the F1s in the fit, and a request can legitimately have 234 individuals in the genotype file and 34 in the association. Verified on BXD.8: excluding BXD1 gives 197 individuals / 33 values; include_parents_f1 with an F1 re-included and a substrain excluded gives 234 / 34. A basket supplies values by name, so a selection applies to a basket-fed mapping the same way: the individuals excluded are simply not in the panel any more, and their basket values are ignored. GenotypeError gains Invalid for this (422) so a caller's typo is not reported as a missing genotype file; the read-only endpoints that can never see one map it the same way for exhaustiveness.
4 daysbnw: module scores as trait values, and BNW's worked examplesPjotr
Two additions that make the endpoint usable rather than merely correct. POST /compute/bnw/features -- from a network to a mappable trait. Every node with at least one parent at the threshold is a module (the node plus its parents), scored per sample by the first principal component of the module's standardised members, sign-corrected so the score rises with the node. Without ?module= the reply is one row per module -- module, members, size, variance_explained -- where variance_explained says whether the score is a fair summary at all. With ?module=NAME it is that module's per-sample scores in the {name, value} shape every mapping endpoint already takes, so the chain runs network -> module -> /compute/rqtl or /compute/gemma with no translation layer. That is what BNW is for in GeneNetwork; the diagram is a by-product. The notes name the construction (a citable module score must say what it is) and repeat the caveat: a module is a correlation structure, not a QTL. GET /compute/bnw/examples and /compute/bnw/examples/{name} -- BNW's own worked examples, for an agent to explore instead of inventing data. The five datasets its tutorial and examples pages present are listed with their sizes, and each is served as a matrix in the values shape /compute/bnw takes, with a ready-to-post action carrying the body. ?columns= restricts the matrix (the spleen example has 14 variables and a caller may want fewer). genotype_column warns that a column is a genotype code: this driver is continuous-only, so it enters the network as a number, the way the web application's GeneNetwork path treats it. Files come from BNW_EXAMPLES_DIR (a gn-bnw checkout's sourcecodes/ example_datasets); a deployment without it answers 404 no_examples. Measured on the dev instance: features over five BXD traits gives two modules (BXD::10001 with BXD::10003 and BXD::10041, 60.4% of variance; BXD::10044 with BXD::10001, 27.2%), and BXD::10001's 19 scores posted straight to /compute/rqtl and /compute/gemma both return 200 -- gemma peaks on chromosome 6. tests/bnw.rs covers the module list, one module's scores in mapping shape, the unknown-module 422, the example list, one example's matrix and its ready-to-post action, and a 404 for a name that does not exist.
4 daysplot/bnw: two SVG figures, drawn the way /plot/gemma draws onePjotr
Post the reply from /compute/bnw back and get an SVG: the same plumbing as /plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/ svg+xml, the plotter's line in x-plot-summary), nothing recomputed. Two figures, because one cannot say what the other can: * the network (default): nodes on a circle, one arrow per edge that survived the threshold, arrow width and label = the edge's posterior, and the below-threshold edges as faint dotted lines -- so a reader sees what was *considered* as well as what survived, which is half of why the posterior table exists; * ?kind=posterior: every candidate edge as a cell in a matrix, dark = high posterior, outlined cells are the ones above the threshold. The diagram cannot show that an edge came within 0.02 of the threshold; this can. Pure-Python SVG, standard library only: no graphviz in the deployment. A circular layout is less pretty than a force-directed one and is deterministic, which matters more for a figure that accompanies a JSON result. Node labels keep the distinguishing tail of a long id, with the full name beside the node. Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the second figure. tests/plot.rs covers both figures plus the 422s (not a BNW result, unknown kind).
5 dayscompute/bnw: a Bayesian network over traits, called out to BNW's driverPjotr
POST /compute/bnw learns a network over a set of variables and returns one row per candidate edge: from, to, posterior, in_structure. Inputs are published traits of a group (group + traits) or an explicit matrix (values: one entry per variable with per-sample values), which covers the case the database cannot -- a basket's columns, a handful of probe sets. BNW's knowledge stays in the gn-bnw repository. GN4 writes a variables x samples matrix, runs `bnw learn` (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR) and parses the reply: nothing about the three C programs, the private work directory or the type line in the input file lives here, so BNW's pipeline can change without a GN4 release. The driver's failures map onto the API's own: bad input 422, a missing driver 502 bnw_not_installed (its exit 77), a failed computation 502 bnw_failed in the driver's words. Decisions worth recording: * parameters are BNW's own defaults -- maxparents 4, k 1, tiers 3, threshold 0.5, read from its create_tiers_gom_part1.php rather than chosen here -- and the reply names the values used, because a network at 0.5 and one at 0.9 are different answers to the same question; * at most 40 variables and at least 3 samples complete across every variable. The first is the structure search's exponential in-degree; the second is BNW modelling a complete multivariate normal, so a sample missing any value is dropped -- and the note says how many survived rather than silently shrinking the panel; * its own gate, always claimed (threshold 0): there is no such thing as a small BNW run, the cost is the search, not the number of individuals; * the notes say plainly that a posterior is not a p-value and that a network is not a QTL, because that is the easiest thing to over-read. tests/bnw.rs compares the endpoint against the answer the BNW *web application* gives for its own six-trait example (the same fixture as the driver's parity check, gn-bnw/test/golden/): the five edges must match, and the notes must name the parameters and carry the caveat. It skips when the driver or its binaries are absent. Not deployed to production.
5 daysdocs: say that project memory is how a session recovers, and where it livesPjotr
The ai-memory store already existed; what was missing was any mention that a reader should use it before starting work, and what they would find. Two concrete things a cold reader needs and cannot reconstruct from a diff: an operational RUNBOOK entry (the Slurm container from gn-machines, its fixed JWT, the environment /compute/gemma/ravanan needs, the four prerequisites that bit the first time, and the fastest test: one chromosome, 14 s), and the "why is this like this" entries behind endpoints that exist because something else was wrong (-pve reporting, the next-link graph, the reference-genome linkage). docs/www/SKILL.md gains section 7a with the three query forms, the topic names that exist (ravanan-slurm, reference-genomes, gemma-pve) and both measured costs; README gains the same note under Development, since that is where a maintainer looks first. States the limit honestly: the store is local, gitignored and per repository -- it survives a cleared session, not a lost machine, which is why every conclusion is also in a commit message and in these docs.
5 daysgemma/ravanan: resolve the same inputs as the direct runner; refuse what it ↵Pjotr
cannot do Two gaps, both invisible until a real request ran against a cluster. The endpoint read only genotype_file, snps_file and trait_values from the request; group, genotype_version and samples were accepted by the shared request type and silently dropped. A /compute/gemma-shaped request (group + samples) therefore arrived as empty paths and empty values and died with "I/O error: No such file or directory (os error 2)" from reading "". Running it exposed the second gap in the same breath: the gate sized the request from trait_values.len(), so a samples-only request was sized 0 and took no concurrency slot at all. resolve_inputs is now one function in routes/gemma.rs, called by both handlers: upload materialised into the pair the engine reads, basket resolved to named values, group prepared with the parent/F1 and empty-genotype rules, named samples aligned by name onto whatever individuals the file has, positional values checked against the panel size. That block was ~250 lines of duplicated rules inside compute_gemma, which is how it drifted in the first place; the direct runner now calls the same function and keeps only its covariates, normalization and gate. The ravanan reply also carries the input notes now (which genotype version, how many values aligned, what was dropped), so a run says what it mapped as well as where the peak is. Refused rather than ignored, because the CWL workflow has no step for either: covariates (422 covariates_not_supported) and normalize (422 normalization_not_supported), each naming /compute/gemma as the endpoint that can do it. A new test asserts both refusals happen before any job is submitted, so it needs no ravanan, ccwl or cluster. Verified against the Slurm container from gn-machines: group: BXD became 40 Slurm jobs (20 kinship + 20 GWA, scattered over the snps file's chromosomes), all completed, 21 055 rows in ~3.5 min. README and docs/ravanan-slurm.md record the four requirements that run turned up, three of them not obvious: ccwl on PATH, a ravanan build with the --slurm-api-endpoint fix, the *server's* HOME writable by the job user inside the cluster (gemma-wrapper locks $HOME/.<hash>.lck there), and GUIX_PACKAGE_PATH for the manifest.