about summary refs log tree commit diff
path: root/src/routes/plot.rs
AgeCommit message (Collapse)Author
38 hourscorrelation matrix: clustered heatmap and its plot HEAD mainPjotr Prins
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix reply and `POST /plot/heatmap` draws it: the r matrix in that order, the dendrogram that produced it, and a diverging scale with blue negative and red positive. A reader can see *why* two traits are neighbours instead of trusting an ordering. Clustering decisions, all recorded because an unrepeatable clustering is worse than none: * distance `1 - r` -- the usual choice for correlations, and it keeps anti-correlated traits apart, which is what a heatmap of correlations should show; * **average linkage (UPGMA)** -- single linkage chains along a gradient and complete linkage makes one flat blob out of noisy traits; average is the compromise GN1's R heatmaps used too; * **deterministic ties and leaf order** (lower index first), so two runs on one matrix produce the same picture and plots stay diffable; * a matrix with a **gap is refused** rather than clustered: an unknown distance treated as 1 would invent a structure, and the reply's note says to drop those traits and ask again. Pairing with the PCA work this is the visual half of a collection: scree plot for how much structure there is, clustered heatmap for where it is. Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back. Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
39 hoursplot: the scree plot of a matrix PCA, which GN2 draws client-sidePjotr Prins
GN2 renders a scree plot with Plotly in the browser, from the explained variance ratios its matrix page computed. This is the same figure as a file: `POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns image/svg+xml, like every other plot endpoint -- the plot is of the numbers that were computed, nothing is recomputed. scripts/plot-pca.py draws a bar per component (its share of the variance, GN2's number), a cumulative line with the 80%-of-variance crossing annotated (what a reader actually uses to choose how many components to keep), the eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps in the matrix -- the one thing GN2 hides by mutating its shared-sample list. `kind=N` on the request caps how many components are drawn, since a 200-trait matrix would otherwise draw 200 bars. Also: the monitor's SVG judge now understands a binary reply (the client hands back content type and size, not the bytes), and `plot-pca` is a check of its own: it asks the matrix for a PCA and posts the reply to the plotter. Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative path, which only resolves when the service's working directory is the checkout). plot-pca and plot-genotypes both ok.
5 daysplot/bnw: two SVG figures, drawn the way /plot/gemma draws onePjotr
Post the reply from /compute/bnw back and get an SVG: the same plumbing as /plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/ svg+xml, the plotter's line in x-plot-summary), nothing recomputed. Two figures, because one cannot say what the other can: * the network (default): nodes on a circle, one arrow per edge that survived the threshold, arrow width and label = the edge's posterior, and the below-threshold edges as faint dotted lines -- so a reader sees what was *considered* as well as what survived, which is half of why the posterior table exists; * ?kind=posterior: every candidate edge as a cell in a matrix, dark = high posterior, outlined cells are the ones above the threshold. The diagram cannot show that an edge came within 0.02 of the threshold; this can. Pure-Python SVG, standard library only: no graphviz in the deployment. A circular layout is less pretty than a force-directed one and is deterministic, which matters more for a figure that accompanies a JSON result. Node labels keep the distinguishing tail of a long id, with the full name beside the node. Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the second figure. tests/plot.rs covers both figures plus the 422s (not a BNW result, unknown kind).
7 daysplot/bulklmm: the 0.05 and 0.01 lines, never 0.10 unless askedPjotr
Checked what the plot actually draws: it has never drawn the alpha 0.10 line -- the plotter matches 0.05 and 0.01 only. What lists all three is the reply's notes ("alpha 0.01: LOD 4.58, alpha 0.05: 3.65, alpha 0.1: 3.50"), which is where the 0.10 came from. Still, the point stands on its own: 0.10 is the loosest bar in the set and drawing it invites reading it as the line to clear. So the defaults are stated and pinned -- 0.05 and 0.01 on the plot, 0.10 only on request (?all_thresholds=true, or an explicit ?thresholds=) -- alongside GEMMA's Bonferroni, which is opt-in for the same reason: a reference line is an annotation, not a result. tests/bulklmm.rs counts the lines: exactly two by default, and no alpha 0.10 label.
7 daysdocs: say that /plot/gemma draws no line unless one is asked forPjotr
The OpenAPI description still promised "a Bonferroni reference line by default", and GEMMA's own suggested plot call said the same. Both now say what the code does, and where the number comes from: the plotter computes 0.05 over the analysed markers -- conservative under linkage and unrelated to relatedness -- and ?thresholds= takes one computed elsewhere, each line labelled with its source. Also corrected in the README's /plot section and in SKILL.md, where the same promise had been copied.
7 daysplot/gemma: no reference line by default; Bonferroni is one more labelled linePjotr
A line on a plot is an annotation, not part of the result, so the default is now no line at all. GEMMA's Bonferroni becomes opt-in like any other: ?bonferroni=true GEMMA's own line, labelled ?thresholds=3.65&threshold_labels=… another engine's, labelled (nothing) just the scan Which means a reader sees exactly the lines someone chose, each with its source in the label -- and the accidental "this plot says significant" impression of a default convention line is gone. The plotter's summary and its <desc> no longer mention Bonferroni unless it was drawn. tests/plot.rs: the plain plot has neither line; with thresholds the two BulkLMM lines are labelled and Bonferroni is absent; with ?bonferroni=true GEMMA's own line appears.
7 daysplot/gemma: draw thresholds computed elsewhere, labelled with their sourcePjotr
GEMMA has no permutation test here, so its plot carries a Bonferroni line. A threshold from /compute/bulklmm belongs to a different null and must not look like GEMMA's, so it is passed in and named: POST /plot/gemma?thresholds=3.50,3.65 &threshold_labels=bulklmm%20alpha%200.10,bulklmm%20alpha%200.05 Each is drawn dashed in the series colour with "label: value" beside the line, with GEMMA's Bonferroni line still there, and the plotter's x-plot-summary names the reference lines it drew. A threshold that is not a positive number is a 422 invalid_threshold rather than a silent omission. /compute/bulklmm's reply offers that href already, with its alpha 0.05 threshold filled in -- so "same values, two models, one picture" is one link from the scan, and the labels keep the two nulls apart. tests/plot.rs: the plain plot has no bulklmm line; with thresholds both are drawn and labelled; a non-numeric threshold is refused.
7 daysbulklmm: exact marker-block parallelism, and what the cores are actually doingPjotr
Reported: a BulkLMM run does not look multicore. It is not a misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint calls -- has no `Threads.@threads` at all; the package threads `bulkscan()` (many traits) and bulkscan_helpers, not scan. The parallelism that was visible before was BLAS (OpenBLAS defaulting to all 96 cores here), which is why `user` time ran 4x `real` while Julia threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s. * The driver splits the markers into independent blocks and scans them on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact: the LOD for each marker is independent given the null variance components, which depend on y and K only. Verified identical -- peak LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks -- and permutation thresholds combine by taking each permutation's maximum across blocks before the quantiles, which is what a genome-wide threshold is. * The parsed matrix and the kinship are cached per panel (read 1.3 s -> 0.4 s, kinship 0.9 s -> 0), because both depend only on the cross. * The summary in `notes` now says what ran: julia threads, blocks, BLAS threads, and the read/scan split. * The hand parser is back: CSV.jl into a DataFrame is an order of magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s). * BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for one scan). Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks 18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of the run and each block re-estimates its variance components. Where the cores pay is bulkscan() -- many traits in one process, threaded over trait blocks -- which is the shape the next endpoint wants; documented in the README rather than left as folklore.
7 daystraits: genotypes at a confirmed peak, and the plot of themPjotr
Returning genotypes is the one thing this API otherwise never does, so it is fenced and the fence is the feature: GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker> * the server re-runs the R/qtl scan for the trait -- the same machinery and gate as /compute/rqtl -- and takes the peak from that run, rather than trusting a client that names a marker; * the marker must lie within ?window= Mb of that peak (default 10), or the answer is 422 not_the_peak quoting the peak's marker, chromosome, Mb and LOD; * the peak must reach ?min_lod= (default 3), or 422 not_a_hit; * only animals that have a value for the trait are listed, and only this marker's calls. Nothing else of the matrix leaves. On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals across A 110 / B 8 / H 74 -- the two a class of eight being exactly the thing worth seeing. POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's `animals` (or its `genotypes` map) with the trait's values and get an SVG with one point per animal, genotype class across, value up, class means marked. Nothing is recomputed -- the plot is of the numbers the client sends, like the other plots. New script scripts/plot-genotypes.py (stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key. tests/peakgenotypes.rs: the peak's calls for animals with values (with the counts agreeing with the rows), the SVG, and a marker on another chromosome refused with the peak quoted. Also keeps /compute/rqtl's error order: naming both a group and a file is "ambiguous" even when the request also forgot its values.
9 daysplot: say which trait a plot is, not just draw itPjotr
Two traits over the same panel really do draw two different pictures -- verified on BXD.4, where all 20 chromosomes' polylines differ and the peaks are LOD 2.93 at chr8 69.4 Mb (mCV24409492) against LOD 4.78 at chr8 95.7 Mb (rs3666069) -- but you could not tell from the title: every plot came out headed **"trait rqtl: LOD along the genome"**, because `/plot/rqtl` labelled it from `?trait=` and nothing else, defaulting to "rqtl". The only visible difference was the peak annotation, and a caller posting the saved response back (which is the documented way) could not know the label was missing. The response now names what it mapped: `/compute/rqtl` writes `Trait: <name>` as its **first** note, and `/plot/rqtl` and `/plot/gemma` take the title's label from that note when `?trait=` was not given. The `POST /plot/rqtl` suggestion in `actions` also carries `?trait=`, so a caller that posts only `data` still gets the right title. * src/routes/rqtl.rs: the `Trait:` note, ahead of the column note; the plot suggestion's href names the trait. * src/routes/plot.rs: `trait_from_document`, used by both plot endpoints. * tests/plot.rs: `two_traits_draw_two_different_plots` -- two canned responses differing only in LOD, asserting each title names its own trait from its own note, each annotates its own peak and not the other's, and every polyline differs. That is the regression this is about: a plot that reused anything between requests would draw the first trait's shape for the second. * tests/{rqtl,phewas}.rs: the plot suggestion is now matched by prefix, with the trait asserted. * README.md, docs/www/ai-instruction.md: where the title comes from, and to post the whole response (or add `?trait=`) if you post only `data`.
11 daysplot: take a posted result, and never re-run the mappingPjotr
The plot endpoints ran the mapping themselves, which meant the picture could come from a different run than the one the caller was looking at, and cost a scan to draw. They now take the mapping's own response -- the JSON /compute/{rqtl,gemma} returned, posted back -- and hand it straight to the plotter, which is what the scripts were written for (`python3 plot-rqtl.py result.json -o out.svg --trait …`). Validation replaces the run: `/plot/rqtl` needs `data.markers` with positional 4+ column rows, `/plot/gemma` needs `data` rows *and* `meta.columns` (its plotter reads the column names to find `p_lrt`), and anything else is a 422 saying "post that response back". `/plot/rqtl` also accepts just the `data` member; GEMMA cannot, and says why. The mapping's `actions` suggestion now describes posting its own response and carries no body -- duplicating a 21 000-row response into a suggestion would double the envelope for nothing, and the caller already holds it. * src/routes/plot.rs: posted result, shape validation, `?trait`/`?title` and `?bonferroni=false` passthrough to the scripts; no compute imports. * src/routes/{rqtl,gemma}.rs: the plot suggestion loses its body. * tests/plot.rs: canned results instead of real mappings -- the tests now need only the plotter, no Rscript or gemma; covers both engines, the bare-`data` form, and the 422 for a body that is a request. * README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: "post the response back", and why GEMMA needs `meta.columns`.
11 daysplot: draw a mapping, and suggest it from the mappingPjotr
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a saved mapping response into an SVG Manhattan panel. They are now endpoints, and the mapping that produces the response suggests them: POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml POST /plot/gemma same body as /compute/gemma -> image/svg+xml Taking the mapping's own body rather than the mapping's output means one call gets you the picture, and the plot cannot describe a run you did not do: the handlers call the compute handlers and hand the resulting envelope to the script, exactly as the scripts' docstrings describe (they read `data.markers`, `data.permutation` and the notes' strain counts). The mapping is re-run, but through the engines' own caches -- seconds on BXD. The reply is an SVG, not the JSON envelope: an image is not JSON, and the plotter's one-line peak summary rides back in `x-plot-summary` so a caller that cannot see the picture still learns where the peak is. Both compute responses gain the matching suggestion: `actions` carries "the same request, plotted" (built from the request that ran, like the permutation suggestion), so an assistant can map, look, and only then decide whether the peak is worth the 100-permutation run. * src/plot.rs: run the configured script over the envelope, read the SVG back, clean up; a missing script is a 502 naming interpreter and script, not a panic. * src/routes/plot.rs: the two handlers (token-gated), delegating to compute_rqtl/compute_gemma so validation and alignment stay in one place; `x-plot-summary` sanitised into a header. * src/routes/{rqtl,gemma}.rs: the plot suggestion, before the permutation one. * src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER. * tests/plot.rs: the suggestion resolves to a complete SVG with a peak summary, and an invalid mapping request is the mapping's own 422. * tests/{rqtl,phewas}.rs: find suggestions by href rather than position. * README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md (new "Plotting": python3 must be on the server's PATH).