| Age | Commit message (Collapse) | Author |
|
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix
reply and `POST /plot/heatmap` draws it: the r matrix in that order, the
dendrogram that produced it, and a diverging scale with blue negative and red
positive. A reader can see *why* two traits are neighbours instead of trusting
an ordering.
Clustering decisions, all recorded because an unrepeatable clustering is worse
than none:
* distance `1 - r` -- the usual choice for correlations, and it keeps
anti-correlated traits apart, which is what a heatmap of correlations should
show;
* **average linkage (UPGMA)** -- single linkage chains along a gradient and
complete linkage makes one flat blob out of noisy traits; average is the
compromise GN1's R heatmaps used too;
* **deterministic ties and leaf order** (lower index first), so two runs on one
matrix produce the same picture and plots stay diffable;
* a matrix with a **gap is refused** rather than clustered: an unknown distance
treated as 1 would invent a structure, and the reply's note says to drop
those traits and ask again.
Pairing with the PCA work this is the visual half of a collection: scree plot
for how much structure there is, clustered heatmap for where it is.
Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation
with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at
all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back.
Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
|
|
GN2 renders a scree plot with Plotly in the browser, from the explained
variance ratios its matrix page computed. This is the same figure as a file:
`POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns
image/svg+xml, like every other plot endpoint -- the plot is of the numbers
that were computed, nothing is recomputed.
scripts/plot-pca.py draws a bar per component (its share of the variance,
GN2's number), a cumulative line with the 80%-of-variance crossing annotated
(what a reader actually uses to choose how many components to keep), the
eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps
in the matrix -- the one thing GN2 hides by mutating its shared-sample list.
`kind=N` on the request caps how many components are drawn, since a
200-trait matrix would otherwise draw 200 bars.
Also: the monitor's SVG judge now understands a binary reply (the client hands
back content type and size, not the bytes), and `plot-pca` is a check of its
own: it asks the matrix for a PCA and posts the reply to the plotter.
Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative
path, which only resolves when the service's working directory is the
checkout). plot-pca and plot-genotypes both ok.
|
|
Post the reply from /compute/bnw back and get an SVG: the same plumbing as
/plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/
svg+xml, the plotter's line in x-plot-summary), nothing recomputed.
Two figures, because one cannot say what the other can:
* the network (default): nodes on a circle, one arrow per edge that survived
the threshold, arrow width and label = the edge's posterior, and the
below-threshold edges as faint dotted lines -- so a reader sees what was
*considered* as well as what survived, which is half of why the posterior
table exists;
* ?kind=posterior: every candidate edge as a cell in a matrix, dark = high
posterior, outlined cells are the ones above the threshold. The diagram
cannot show that an edge came within 0.02 of the threshold; this can.
Pure-Python SVG, standard library only: no graphviz in the deployment. A
circular layout is less pretty than a force-directed one and is deterministic,
which matters more for a figure that accompanies a JSON result. Node labels
keep the distinguishing tail of a long id, with the full name beside the node.
Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw
reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the
strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the
second figure. tests/plot.rs covers both figures plus the 422s (not a BNW
result, unknown kind).
|
|
Checked what the plot actually draws: it has never drawn the alpha 0.10
line -- the plotter matches 0.05 and 0.01 only. What lists all three is the
reply's notes ("alpha 0.01: LOD 4.58, alpha 0.05: 3.65, alpha 0.1: 3.50"),
which is where the 0.10 came from.
Still, the point stands on its own: 0.10 is the loosest bar in the set and
drawing it invites reading it as the line to clear. So the defaults are
stated and pinned -- 0.05 and 0.01 on the plot, 0.10 only on request
(?all_thresholds=true, or an explicit ?thresholds=) -- alongside GEMMA's
Bonferroni, which is opt-in for the same reason: a reference line is an
annotation, not a result.
tests/bulklmm.rs counts the lines: exactly two by default, and no
alpha 0.10 label.
|
|
The OpenAPI description still promised "a Bonferroni reference line by
default", and GEMMA's own suggested plot call said the same. Both now say
what the code does, and where the number comes from: the plotter computes
0.05 over the analysed markers -- conservative under linkage and unrelated
to relatedness -- and ?thresholds= takes one computed elsewhere, each line
labelled with its source.
Also corrected in the README's /plot section and in SKILL.md, where the
same promise had been copied.
|
|
A line on a plot is an annotation, not part of the result, so the default
is now no line at all. GEMMA's Bonferroni becomes opt-in like any other:
?bonferroni=true GEMMA's own line, labelled
?thresholds=3.65&threshold_labels=… another engine's, labelled
(nothing) just the scan
Which means a reader sees exactly the lines someone chose, each with its
source in the label -- and the accidental "this plot says significant"
impression of a default convention line is gone. The plotter's summary and
its <desc> no longer mention Bonferroni unless it was drawn.
tests/plot.rs: the plain plot has neither line; with thresholds the two
BulkLMM lines are labelled and Bonferroni is absent; with ?bonferroni=true
GEMMA's own line appears.
|
|
GEMMA has no permutation test here, so its plot carries a Bonferroni line.
A threshold from /compute/bulklmm belongs to a different null and must not
look like GEMMA's, so it is passed in and named:
POST /plot/gemma?thresholds=3.50,3.65
&threshold_labels=bulklmm%20alpha%200.10,bulklmm%20alpha%200.05
Each is drawn dashed in the series colour with "label: value" beside the
line, with GEMMA's Bonferroni line still there, and the plotter's
x-plot-summary names the reference lines it drew. A threshold that is not
a positive number is a 422 invalid_threshold rather than a silent omission.
/compute/bulklmm's reply offers that href already, with its alpha 0.05
threshold filled in -- so "same values, two models, one picture" is one
link from the scan, and the labels keep the two nulls apart.
tests/plot.rs: the plain plot has no bulklmm line; with thresholds both
are drawn and labelled; a non-numeric threshold is refused.
|
|
Reported: a BulkLMM run does not look multicore. It is not a
misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint
calls -- has no `Threads.@threads` at all; the package threads
`bulkscan()` (many traits) and bulkscan_helpers, not scan. The
parallelism that was visible before was BLAS (OpenBLAS defaulting to all
96 cores here), which is why `user` time ran 4x `real` while Julia
threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s.
* The driver splits the markers into independent blocks and scans them
on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact:
the LOD for each marker is independent given the null variance
components, which depend on y and K only. Verified identical -- peak
LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks --
and permutation thresholds combine by taking each permutation's
maximum across blocks before the quantiles, which is what a
genome-wide threshold is.
* The parsed matrix and the kinship are cached per panel (read 1.3 s ->
0.4 s, kinship 0.9 s -> 0), because both depend only on the cross.
* The summary in `notes` now says what ran: julia threads, blocks, BLAS
threads, and the read/scan split.
* The hand parser is back: CSV.jl into a DataFrame is an order of
magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s).
* BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for
one scan).
Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks
18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of
the run and each block re-estimates its variance components. Where the
cores pay is bulkscan() -- many traits in one process, threaded over
trait blocks -- which is the shape the next endpoint wants; documented in
the README rather than left as folklore.
|
|
Returning genotypes is the one thing this API otherwise never does, so it
is fenced and the fence is the feature:
GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker>
* the server re-runs the R/qtl scan for the trait -- the same machinery
and gate as /compute/rqtl -- and takes the peak from that run, rather
than trusting a client that names a marker;
* the marker must lie within ?window= Mb of that peak (default 10), or
the answer is 422 not_the_peak quoting the peak's marker, chromosome,
Mb and LOD;
* the peak must reach ?min_lod= (default 3), or 422 not_a_hit;
* only animals that have a value for the trait are listed, and only
this marker's calls. Nothing else of the matrix leaves.
On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals
across A 110 / B 8 / H 74 -- the two a class of eight being exactly the
thing worth seeing.
POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's
`animals` (or its `genotypes` map) with the trait's values and get an SVG
with one point per animal, genotype class across, value up, class means
marked. Nothing is recomputed -- the plot is of the numbers the client
sends, like the other plots. New script scripts/plot-genotypes.py
(stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key.
tests/peakgenotypes.rs: the peak's calls for animals with values (with the
counts agreeing with the rows), the SVG, and a marker on another
chromosome refused with the peak quoted.
Also keeps /compute/rqtl's error order: naming both a group and a file is
"ambiguous" even when the request also forgot its values.
|
|
Two traits over the same panel really do draw two different pictures --
verified on BXD.4, where all 20 chromosomes' polylines differ and the peaks
are LOD 2.93 at chr8 69.4 Mb (mCV24409492) against LOD 4.78 at chr8
95.7 Mb (rs3666069) -- but you could not tell from the title: every plot
came out headed **"trait rqtl: LOD along the genome"**, because
`/plot/rqtl` labelled it from `?trait=` and nothing else, defaulting to
"rqtl". The only visible difference was the peak annotation, and a caller
posting the saved response back (which is the documented way) could not
know the label was missing.
The response now names what it mapped: `/compute/rqtl` writes `Trait:
<name>` as its **first** note, and `/plot/rqtl` and `/plot/gemma` take the
title's label from that note when `?trait=` was not given. The
`POST /plot/rqtl` suggestion in `actions` also carries `?trait=`, so a
caller that posts only `data` still gets the right title.
* src/routes/rqtl.rs: the `Trait:` note, ahead of the column note; the
plot suggestion's href names the trait.
* src/routes/plot.rs: `trait_from_document`, used by both plot endpoints.
* tests/plot.rs: `two_traits_draw_two_different_plots` -- two canned
responses differing only in LOD, asserting each title names its own
trait from its own note, each annotates its own peak and not the
other's, and every polyline differs. That is the regression this is
about: a plot that reused anything between requests would draw the
first trait's shape for the second.
* tests/{rqtl,phewas}.rs: the plot suggestion is now matched by prefix,
with the trait asserted.
* README.md, docs/www/ai-instruction.md: where the title comes from, and
to post the whole response (or add `?trait=`) if you post only `data`.
|
|
The plot endpoints ran the mapping themselves, which meant the picture
could come from a different run than the one the caller was looking at,
and cost a scan to draw. They now take the mapping's own response -- the
JSON /compute/{rqtl,gemma} returned, posted back -- and hand it straight
to the plotter, which is what the scripts were written for
(`python3 plot-rqtl.py result.json -o out.svg --trait …`).
Validation replaces the run: `/plot/rqtl` needs `data.markers` with
positional 4+ column rows, `/plot/gemma` needs `data` rows *and*
`meta.columns` (its plotter reads the column names to find `p_lrt`), and
anything else is a 422 saying "post that response back". `/plot/rqtl`
also accepts just the `data` member; GEMMA cannot, and says why.
The mapping's `actions` suggestion now describes posting its own response
and carries no body -- duplicating a 21 000-row response into a
suggestion would double the envelope for nothing, and the caller already
holds it.
* src/routes/plot.rs: posted result, shape validation, `?trait`/`?title`
and `?bonferroni=false` passthrough to the scripts; no compute imports.
* src/routes/{rqtl,gemma}.rs: the plot suggestion loses its body.
* tests/plot.rs: canned results instead of real mappings -- the tests now
need only the plotter, no Rscript or gemma; covers both engines, the
bare-`data` form, and the 422 for a body that is a request.
* README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md:
"post the response back", and why GEMMA needs `meta.columns`.
|
|
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a
saved mapping response into an SVG Manhattan panel. They are now
endpoints, and the mapping that produces the response suggests them:
POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml
POST /plot/gemma same body as /compute/gemma -> image/svg+xml
Taking the mapping's own body rather than the mapping's output means one
call gets you the picture, and the plot cannot describe a run you did not
do: the handlers call the compute handlers and hand the resulting
envelope to the script, exactly as the scripts' docstrings describe
(they read `data.markers`, `data.permutation` and the notes' strain
counts). The mapping is re-run, but through the engines' own caches --
seconds on BXD.
The reply is an SVG, not the JSON envelope: an image is not JSON, and the
plotter's one-line peak summary rides back in `x-plot-summary` so a
caller that cannot see the picture still learns where the peak is.
Both compute responses gain the matching suggestion: `actions` carries
"the same request, plotted" (built from the request that ran, like the
permutation suggestion), so an assistant can map, look, and only then
decide whether the peak is worth the 100-permutation run.
* src/plot.rs: run the configured script over the envelope, read the
SVG back, clean up; a missing script is a 502 naming interpreter and
script, not a panic.
* src/routes/plot.rs: the two handlers (token-gated), delegating to
compute_rqtl/compute_gemma so validation and alignment stay in one
place; `x-plot-summary` sanitised into a header.
* src/routes/{rqtl,gemma}.rs: the plot suggestion, before the
permutation one.
* src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER.
* tests/plot.rs: the suggestion resolves to a complete SVG with a peak
summary, and an invalid mapping request is the mapping's own 422.
* tests/{rqtl,phewas}.rs: find suggestions by href rather than position.
* README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md
(new "Plotting": python3 must be on the server's PATH).
|