| Age | Commit message (Collapse) | Author |
|
Three local patches, the third of which is the reason a 20-chromosome scatter
now runs 20-wide instead of 1-wide: {"job": ...} versus {"jobs": [...]},
which is slurmrestd's heterogeneous-job form. With the node declaring 30 CPUs
(cold 24.7 s) and the submit fixed (cold 9.4 s), ravanan is within 3x of
gemma-wrapper's 3.1 s for the same rows instead of 33x.
|
|
A warm run answers from caches the previous binary filled, so it can pass
while the cold path -- what a new caller's first request actually does -- is
broken. `--cold` sends each cached engine a payload with one sample value
nudged by a thousandth, then repeats the original: every engine here keys its
cache on content (gemma-wrapper's hash, ravanan's interned inputs,
rqtl/qtl2/bulklmm's own), so the first is a miss and the second a hit. Both
runs are judged; the line reports "cold X / warm Y".
Measured on production with the 30-sample fixture:
compute-ravanan 102.0 s cold / 3.7 s warm
compute-gemma 3.1 s cold / 2.0 s warm
compute-rqtl 7.5 s cold / 7.3 s warm
compute-qtl2 3.3 s cold / 3.3 s warm
compute-fitqtl 6.7 s cold / 6.9 s warm
compute-bulklmm 19.6 s cold / 19.0 s warm
ravanan is the one where the cache is the difference between minutes and
seconds; the rest are compute-bound at this size, and the honest thing is to
print the pair rather than assert a speedup. A check whose payload cannot be
perturbed skips with that reason instead of reporting two warm runs as cold
and warm.
The operator checklist gains the cold run as part of deploying, with both
costs named: ravanan's cold run takes minutes, and every cold run adds cache
entries.
|
|
monitor run
`POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}`
references -- a collection, as GN2's matrix form posts it, which spans
datasets -- and answers an N x N matrix of r, p and n.
The five decisions that are not obvious, each recorded where a reader will
look for them:
* **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to
Spearman mid-row once a Pearson r passes 0.999, which the self-pair always
does, so its matrix is asymmetric by construction. Here the method is
chosen once and r[i][j] == r[j][i] exactly.
* **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine
calls, not N^2.
* **The diagonal is by construction**: r = 1, p = null, n = the trait's own
count, so half the traits cannot come back as 1.0000000000000002 with a
NaN p.
* **Per-cell n is its own matrix**, and a cell under `min_overlap` (default
6) is null in r and p while its n still says how many points there were.
GN2 accepts 2, where |r| = 1 by definition.
* **The sample axis is the first trait's group** (#106), resolved once and
stated in the reply with the number of individuals it kept.
Fetches are one query per dataset (`get_dataset_values_for_traits`,
`get_publish_values_for_traits`), not one per trait: a matrix chooses a
handful out of twenty thousand. Capped at 1 000 traits, because the cost is
quadratic in the correlations and in the reply. No PCA -- the r matrix has
to be trusted on its own first.
Verified twice: cells agree with scipy to the last bit (BXDPublish
10001/10002/10005: 0.6756270426459695, 0.6950450737755965,
0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964,
0.8819879006700073), and a cell equals what the row endpoint says for the
same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix,
`min_overlap` above every cell's n, an unknown trait, `bicor`, and no
traits at all.
The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier
all`, and says why: `/version` can report a commit and a reachable database
while every engine that shells out is dead. The monitor gained a matrix
check whose judge enforces square, symmetric and a unit diagonal.
|
|
`--only` needs exact check names, which is no help when what you know is the
dependency ("julia", "Rscript", "slurmrestd"). `--filter` matches a substring
against the check's name, the sentence describing what it needs, and its
hint -- so `--filter julia` finds compute-bulklmm even though its name says
nothing about Julia. It selects tiers like `--only` does, and says so when
nothing matches *because* discovery has not run (the compute checks do not
exist until a group with samples is known).
Also un-ignores bin/: the monitor is source, and an ops tool that is not
versioned is the problem ai-memory #38 already named. Bytecode stays out.
Co-report: the whole compute tier is green now (9 ok, 0 failed), including
BulkLMM -- the recipe and the depot quirk are in the operator checklist.
|
|
work yet
Writing down what the monitor found and what fixed it, because it is config
that lives outside the repository and is therefore easy to lose at the next
deploy: with only the cache dirs set, every engine that shells out answers
502 while the whole read surface and search look healthy.
Measured on production after the fix (bin/monitor-gn4 --tier compute):
8 ok, 1 failed -- correlation, gemma (21 055 rows), rqtl, qtl2 (20 453),
fitqtl, ravanan (21 055 rows), bnw, bnw-examples. Before: 1 ok, 7 failed,
1 busy.
The remaining failure is BulkLMM and it is not a config line: the Julia
project is registry-based and no depot on this box has its packages, so
it stops at "Run Pkg.instantiate()". Populating a depot downloads from the
Julia registry, which should be a decision rather than a side effect.
|
|
A survey of the three clones side by side (genenetwork1, genenetwork2,
genenetwork3) for functionality that would fit a JSON API, ordered by what
it would cost to migrate rather than by which repo it came from, in three
tiers: S (hours to a day), M (days), L/XL (weeks or a new concept).
Each row says which of GN1/GN2/GN3 carries the feature -- yes, part or no,
established by reading their code -- so a row is not mistaken for something
nobody has ever built. Nine rows are S, eight M, eight L/XL; the S tier is
almost entirely metadata, which is exactly the pattern the size evidence
explains: ProbeSetData 5 314 694 464 rows, LCorrRamin3 127 368 780, SnpAll
79 534 772 and GeneRIF 8 425 all in db_webqtl, so every value-shaped feature
is a query-design problem while every metadata feature is small.
Also lists what genenetwork4 already covers, the recommended order (matrix,
then tissue and literature correlation, then pagination and the dropdown
tree, then the sample-matrix and genotype exports, then partial
correlation), and where each feature lives in each clone.
ASCII only, aligned so the tables read as tables in a monospace view.
|
|
container)
The layout, the restart command, where mutable state lives, and the two
mistakes that cost time today: the CWL comes from the checkout rather than
the store, and only two of the four ravanan 0.2.0 builds on this box
convert --slurm-api-endpoint to a URI -- the others die inside
slurm-api.scm with struct-vtable on a string, after staging every input,
which reads like a Slurm fault rather than an argument one.
|
|
Synchronous, no redis, no job: one POST computes and answers. Use the
correlation_rust engine in-process (already packaged, rev 473ad82), reuse
its maths rather than rewriting the p-value, and implement only the sort
and top-N. Minimum overlap 6 (gn3/python, not the engine's 4), pearson
and spearman only, the engine's exact |r|-descending/NaN-last ordering,
top-N as a request option applied after the sort, and the sample axis
first.
Verification is against the live genenetwork.org mapping page: it answers
HTTP 200 in ~31 s, so the earlier "-m 20 curl" made it look unreachable
when it is merely slow. Budget ~31 s per reference page in the parity
loop.
|
|
Answers the question that blocks duplicating the correlation_rust path:
what is actually correlated. Four kinds over three axes -- sample (one
trait vs every trait of a dataset, over individuals), tissue (gene vs gene
over 26-32 tissue systems), literature (a table lookup in LCorrRamin3, no
n at all) and partial (conditioned, GN3 only, async).
Records the engine's contract as it is: JSON in, CSV out, one trait per
line positionally aligned to x_vals, non-floats dropped pairwise, rows
under 4 pairs refused, output sorted by |r| descending with NaNs last,
top-N applied afterwards in Python, and no bicor in the engine at all --
which matters because GN3's Python path uses a minimum of 6 and the engine
uses 4.
Measured read-only against db_webqtl: ProbeSetData 5 314 694 464 rows,
PublishData 3 288 638, TissueProbeSetData 90 562 DataIds with 26-32 values
each (avg 29.0), LCorrRamin3 127 368 780. Hence no path loads a dataset
whole, and the value fetch -- not the arithmetic -- is the cost.
Ends with the two shapes a GN4 integration can take (thin: keep the file
dance; direct: reuse parser::parse_rows_with_names and Pearson/Spearman in
memory) and the four semantics that must be decided either way.
|
|
Measured against the slurm container, 20 chromosomes, 40 jobs: baseline
200/21 055 rows in 205.5 s; slurmrestd unreachable -> 502 in 3.7 s;
container killed 45 s into a request -> 502 in 50.2 s with 0 rows (no
partial results); the same request after Slurm returns -> 200 in 163.3 s,
i.e. the retry reuses the steps that had finished, because ravanan's store
lives in RAVANAN_CACHE_DIR and not in Slurm.
So: no automatic failover, no fallback to single-machine, nothing sticky,
and a retry that is cheaper than cold. The 502's detail is ravanan's Guile
backtrace, so the one useful line sits at its end; a pre-flight
/slurm/v0.0.41/ping would make that a 503 naming the endpoint, and there
is no timeout on the ravanan child for a Slurm that accepts connections
and never runs the job -- both recorded as gaps.
|
|
Ravanan's reuse key is (script, inputs) and both live in its --store under
RAVANAN_CACHE_DIR, so the cache's lifetime is that directory's, not the
container's. Measured: 13.6 s cold, then 3.6 s for the identical request
after restarting the container, with the store unchanged at 14 files --
Slurm's own spool state is lost and irrelevant to reuse.
The default cache dir is under /tmp, which is neither durable nor visible
inside a container that bind-mounts one host path, so a job cannot read
the interned inputs. The reply now carries a note when the cache dir is
under /tmp, naming the fix, rather than leaving it to be discovered as a
slow or failing run.
docs/ravanan-slurm.md gains a "What survives a restart" table (ravanan
store, scratch, token registry, gemma-wrapper cache, Slurm state) and a
path checklist; the README points at it.
|
|
The access log said who (`user`) but not with what: with short tokens a
request can be made by a shared credential, so "which person" and "which
credential" are now separate fields -- `scheme` ("jwt" | "opaque") and,
for an opaque token, `token` (its registry handle, e.g.
`agent-ravanan-3`). Never the token or its hash, and the path still
carries no query string.
The line also gains `ts` (RFC 3339 UTC), so a redirected stderr file is
self-describing rather than relying on journald's prefix. Hand-rolled
civil-from-days rather than a new crate: one format, and no Cargo.lock
drift.
scripts/gn4-log.py reads such a log and shows it as a table with the
handle joined against tokens.sqlite: the subject as the registry has it
now (flagging a handle whose subject has been repointed), and the token's
status today -- [revoked], [expired], [unknown] -- which is how "an agent
was working at 14:02" is told apart from "an agent was working at 14:02
with a credential revoked since". It reads the registry read-only, passes
non-JSON lines through in table mode (the process log interleaves
backtraces), and offers --json and --summary for jq and for counting.
CallerSlot now carries the whole Caller (scheme, id, subject) rather than
just the subject string.
|
|
A second credential format beside gn-auth's JWT, for agents and scripts:
`gn4ai_` + 56 body characters + 2 checksum characters = 64, from an
alphabet with no look-alikes, so a typo is refused before the registry is
consulted. Opaque by design -- nothing is stored in the token; every fact
lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode
0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row,
takes effect on the next request, and a token in a log line reveals
nothing; lose the registry and every short token stops working.
Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and
GET /tokens/who are behind a router layer that refuses any caller whose
TCP peer is not 127.x.x.x/::1, and any request claiming to come through a
proxy. A layer rather than a per-handler check, so nothing of a remote
request is parsed before it is refused; there is no configuration switch,
because minting credentials must never be reachable from the network.
A short token reads public data only: every token-only endpoint, and
`#[gnauth(secured)]` endpoints for public rows; a confidential dataset is
refused with a 403 that names the JWT route, since gn-auth cannot validate
an opaque token (the on-the-fly-bridge and registry-grant alternatives
were both rejected for v1).
scripts/gn4-token.py create|list|revoke|who drives the routes, so there is
one implementation of the format and the revocation rules; the token is
printed exactly once.
Tests: the localhost guard over all four routes (remote peer, no peer,
proxied header), and mint -> authenticate -> who -> revoke -> 401.
|
|
docs/www/SKILL.md: the workflow example now calls /compute/gemma/ravanan,
and a new "Which GEMMA endpoint" section says when not to: Slurm down or
submission failing, the user asks, a field ravanan refuses (covariates,
normalization), or verifying results -- where the wrapper endpoint is the
reference and scripts/gemma-compare.py is the check.
docs/dev/SKILL.md: the same rule from the developer side, in Testing --
verify mapping changes by sending one body to both endpoints and never
'fix' a difference by switching endpoints, plus the reminder that the
wrapper's cache only hits from 5eb53fa on, so a stale gemma-wrapper on
PATH looks like a broken cache.
|
|
The proxy logs the caller through x-gn4-user, but the process log -- the one an
operator tails while a job runs -- named nobody, so "who started this GEMMA
run" meant going to the nginx log or guessing. genenetwork4 now writes one
compact JSON line per request on stderr:
{"event":"request","method":"GET","path":"/groups/BXD","status":200,
"took_ms":10,"request_id":"…","user":"zachs"}
Decisions in it: `user` is absent, not "-", when no token was verified (a dash
is a value a reader can mistake for a user name); the path is logged without
its query string, because a query can carry a credential and has no business
in a log; the token value itself is never logged, only its subject; and the
line carries the `request_id`, so it sits beside the existing `event: error`
lines for 5xx and one grep joins a failure to the request that caused it.
`GN4_ACCESS_LOG=false` turns it off where the proxy's log is the record of
truth.
Verified on a probe instance: with the test token the line says
user "gn4-test-stub", with zachs's token user "zachs", and an unauthenticated
401 has no user field at all.
docs/www/operator-checklist.md claimed "genenetwork4 does not log the token's
sub" -- true until now, which is exactly the kind of sentence that rots; it is
replaced by the line above, what it contains and what it deliberately omits.
README's x-gn4-user paragraph gains the same, next to the identity it already
described.
|
|
The API already computes every reply's options -- `next` are GETs that answer
200, `actions` are the POSTs to run next -- and a caller that reads them
learned nothing extra from being told "you could now do X" in prose. What was
missing was the instruction: an answer that stops at the result leaves the
reader to work out the graph.
docs/www/SKILL.md gains "Always offer the next step": end every answer with
ONE sentence built from that reply's own next/actions -- after a compute run, a
lookup, a plot, an upload, a basket edit. Prefer the action href over a
hand-built call. A refusal is an answer too: its next or detail names the
alternative (bulklmm for a GEMMA permutation, /compute/gemma for a covariate
fit). Do not stack suggestions; one next plus at most one genuinely different
alternative. "Skip the suggestions" / "just the result" turns it off for the
session -- it is the sentence that is disabled, not the content. And never
propose a step the API cannot do: if nothing in next/actions fits, say so.
README states the same bargain where the next-link promise is described: the
server's half is that the suggestion is always reachable, the assistant's half
is that it is one sentence and it is a link the reply actually carries.
|
|
gemma-wrapper was doing two things ravanan now does or GEMMA already
does: the parallel fan-out (ravanan's CWL scatter replaced it long ago,
which is why every job passed --no-parallel) and LOCO orchestration --
gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and
takes the annotation file it needs. What was left was a content-hash K
cache and a --json record with results packed into .tar.xz archives.
So cwl/gemma/gemma-loco.scm now runs two gemma invocations per
chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix>
-lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat,
deterministic names. The kinship matrix travels from one step to the
other as a CWL File output, and each job's .assoc.txt and .log.txt are
the workflow's outputs -- so compute_ravanan reads files instead of
parsing a wrapper JSON record and unpacking archives, and the pve note
comes straight out of the jobs' logs.
Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows --
636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to
end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper
also needed Ruby and pfff on PATH, and named its outputs with the
extension doubled (...1.assoc.txt.assoc.txt).
/compute/gemma keeps gemma-wrapper, deliberately: that is what makes
the two endpoints comparable rather than two copies of one stack.
scripts/gemma-compare.py runs the same body through both and reports
row-level differences and timings.
The gemma-wrapper content-hash K cache is the one thing given up;
a repeat request recomputes (271 ms of kinship on this fixture).
|
|
Parents and F1s are excluded by default (as in GeneNetwork2/3), with
include_parents_f1 to keep them all. That boolean was not enough for what was
asked -- "include these, drop that substrain" -- and the exclusion list itself
is data (parents_and_f1s.json, seven groups) that no request can reach. F1
names also differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH,
C57BL/6JxBXD037F1 for the intercross F1s), so pattern matching is the wrong
tool.
Every group-mapping endpoint -- /compute/gemma, /compute/gemma/ravanan,
/compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm -- now takes
"individuals": {"include": [...], "exclude": [...]}
applied after the parent/F1 rule, exclude winning over include. An unknown name
is a 422 invalid_individuals whose message lists the panel: a silently dropped
animal changes a kinship matrix without anyone noticing, so the failure has to
be loud. genotypes::select_panel is the one implementation, and it returns what
it did (dropped parents/F1s, re-included by name, excluded by name) so the
Genotype note can say it: "197 individuals, 38 parents/F1s dropped; 1 excluded
by name".
The note now also spells out the distinction that makes this option
treacherous: the kinship uses *every* individual in the genotype file, the
association uses only those with a trait value -- GEMMA reads no phenotypes for
the kinship. So "include the F1s" changes the relatedness structure without
putting the F1s in the fit, and a request can legitimately have 234 individuals
in the genotype file and 34 in the association. Verified on BXD.8: excluding
BXD1 gives 197 individuals / 33 values; include_parents_f1 with an F1
re-included and a substrain excluded gives 234 / 34.
A basket supplies values by name, so a selection applies to a basket-fed
mapping the same way: the individuals excluded are simply not in the panel any
more, and their basket values are ignored.
GenotypeError gains Invalid for this (422) so a caller's typo is not reported
as a missing genotype file; the read-only endpoints that can never see one map
it the same way for exhaustiveness.
|
|
Two additions that make the endpoint usable rather than merely correct.
POST /compute/bnw/features -- from a network to a mappable trait. Every node
with at least one parent at the threshold is a module (the node plus its
parents), scored per sample by the first principal component of the module's
standardised members, sign-corrected so the score rises with the node. Without
?module= the reply is one row per module -- module, members, size,
variance_explained -- where variance_explained says whether the score is a fair
summary at all. With ?module=NAME it is that module's per-sample scores in the
{name, value} shape every mapping endpoint already takes, so the chain runs
network -> module -> /compute/rqtl or /compute/gemma with no translation layer.
That is what BNW is for in GeneNetwork; the diagram is a by-product. The notes
name the construction (a citable module score must say what it is) and repeat
the caveat: a module is a correlation structure, not a QTL.
GET /compute/bnw/examples and /compute/bnw/examples/{name} -- BNW's own worked
examples, for an agent to explore instead of inventing data. The five datasets
its tutorial and examples pages present are listed with their sizes, and each is
served as a matrix in the values shape /compute/bnw takes, with a ready-to-post
action carrying the body. ?columns= restricts the matrix (the spleen example
has 14 variables and a caller may want fewer). genotype_column warns that a
column is a genotype code: this driver is continuous-only, so it enters the
network as a number, the way the web application's GeneNetwork path treats it.
Files come from BNW_EXAMPLES_DIR (a gn-bnw checkout's sourcecodes/
example_datasets); a deployment without it answers 404 no_examples.
Measured on the dev instance: features over five BXD traits gives two modules
(BXD::10001 with BXD::10003 and BXD::10041, 60.4% of variance; BXD::10044 with
BXD::10001, 27.2%), and BXD::10001's 19 scores posted straight to /compute/rqtl
and /compute/gemma both return 200 -- gemma peaks on chromosome 6. tests/bnw.rs
covers the module list, one module's scores in mapping shape, the unknown-module
422, the example list, one example's matrix and its ready-to-post action, and a
404 for a name that does not exist.
|
|
Post the reply from /compute/bnw back and get an SVG: the same plumbing as
/plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/
svg+xml, the plotter's line in x-plot-summary), nothing recomputed.
Two figures, because one cannot say what the other can:
* the network (default): nodes on a circle, one arrow per edge that survived
the threshold, arrow width and label = the edge's posterior, and the
below-threshold edges as faint dotted lines -- so a reader sees what was
*considered* as well as what survived, which is half of why the posterior
table exists;
* ?kind=posterior: every candidate edge as a cell in a matrix, dark = high
posterior, outlined cells are the ones above the threshold. The diagram
cannot show that an edge came within 0.02 of the threshold; this can.
Pure-Python SVG, standard library only: no graphviz in the deployment. A
circular layout is less pretty than a force-directed one and is deterministic,
which matters more for a figure that accompanies a JSON result. Node labels
keep the distinguishing tail of a long id, with the full name beside the node.
Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw
reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the
strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the
second figure. tests/plot.rs covers both figures plus the 422s (not a BNW
result, unknown kind).
|
|
POST /compute/bnw learns a network over a set of variables and returns one row
per candidate edge: from, to, posterior, in_structure. Inputs are published
traits of a group (group + traits) or an explicit matrix (values: one entry per
variable with per-sample values), which covers the case the database cannot --
a basket's columns, a handful of probe sets.
BNW's knowledge stays in the gn-bnw repository. GN4 writes a variables x
samples matrix, runs `bnw learn` (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR) and
parses the reply: nothing about the three C programs, the private work
directory or the type line in the input file lives here, so BNW's pipeline can
change without a GN4 release. The driver's failures map onto the API's own:
bad input 422, a missing driver 502 bnw_not_installed (its exit 77), a failed
computation 502 bnw_failed in the driver's words.
Decisions worth recording:
* parameters are BNW's own defaults -- maxparents 4, k 1, tiers 3, threshold
0.5, read from its create_tiers_gom_part1.php rather than chosen here --
and the reply names the values used, because a network at 0.5 and one at
0.9 are different answers to the same question;
* at most 40 variables and at least 3 samples complete across every variable.
The first is the structure search's exponential in-degree; the second is
BNW modelling a complete multivariate normal, so a sample missing any value
is dropped -- and the note says how many survived rather than silently
shrinking the panel;
* its own gate, always claimed (threshold 0): there is no such thing as a
small BNW run, the cost is the search, not the number of individuals;
* the notes say plainly that a posterior is not a p-value and that a network
is not a QTL, because that is the easiest thing to over-read.
tests/bnw.rs compares the endpoint against the answer the BNW *web
application* gives for its own six-trait example (the same fixture as the
driver's parity check, gn-bnw/test/golden/): the five edges must match, and the
notes must name the parameters and carry the caveat. It skips when the driver
or its binaries are absent.
Not deployed to production.
|
|
The ai-memory store already existed; what was missing was any mention that a
reader should use it before starting work, and what they would find. Two
concrete things a cold reader needs and cannot reconstruct from a diff: an
operational RUNBOOK entry (the Slurm container from gn-machines, its fixed JWT,
the environment /compute/gemma/ravanan needs, the four prerequisites that bit
the first time, and the fastest test: one chromosome, 14 s), and the "why is
this like this" entries behind endpoints that exist because something else was
wrong (-pve reporting, the next-link graph, the reference-genome linkage).
docs/www/SKILL.md gains section 7a with the three query forms, the topic names
that exist (ravanan-slurm, reference-genomes, gemma-pve) and both measured
costs; README gains the same note under Development, since that is where a
maintainer looks first.
States the limit honestly: the store is local, gitignored and per repository --
it survives a cleared session, not a lost machine, which is why every
conclusion is also in a commit message and in these docs.
|
|
cannot do
Two gaps, both invisible until a real request ran against a cluster.
The endpoint read only genotype_file, snps_file and trait_values from the
request; group, genotype_version and samples were accepted by the shared
request type and silently dropped. A /compute/gemma-shaped request
(group + samples) therefore arrived as empty paths and empty values and
died with "I/O error: No such file or directory (os error 2)" from
reading "". Running it exposed the second gap in the same breath: the
gate sized the request from trait_values.len(), so a samples-only request
was sized 0 and took no concurrency slot at all.
resolve_inputs is now one function in routes/gemma.rs, called by both
handlers: upload materialised into the pair the engine reads, basket
resolved to named values, group prepared with the parent/F1 and
empty-genotype rules, named samples aligned by name onto whatever
individuals the file has, positional values checked against the panel
size. That block was ~250 lines of duplicated rules inside compute_gemma,
which is how it drifted in the first place; the direct runner now calls
the same function and keeps only its covariates, normalization and gate.
The ravanan reply also carries the input notes now (which genotype
version, how many values aligned, what was dropped), so a run says what it
mapped as well as where the peak is.
Refused rather than ignored, because the CWL workflow has no step for
either: covariates (422 covariates_not_supported) and normalize (422
normalization_not_supported), each naming /compute/gemma as the endpoint
that can do it. A new test asserts both refusals happen before any job is
submitted, so it needs no ravanan, ccwl or cluster.
Verified against the Slurm container from gn-machines: group: BXD became
40 Slurm jobs (20 kinship + 20 GWA, scattered over the snps file's
chromosomes), all completed, 21 055 rows in ~3.5 min. README and
docs/ravanan-slurm.md record the four requirements that run turned up,
three of them not obvious: ccwl on PATH, a ravanan build with the
--slurm-api-endpoint fix, the *server's* HOME writable by the job user
inside the cluster (gemma-wrapper locks $HOME/.<hash>.lck there), and
GUIX_PACKAGE_PATH for the manifest.
|
|
Two things a reader needs before quoting a position from this API, both
measured against db_webqtl rather than inferred from the schema:
- No table stores an assembly id. The only labels are prose in
Species.MenuName: human hg19, mouse mm10, rat "mRatBN7.2/rn7", fly dm6,
nematode ce10, arabidopsis araTha1, medaka Oryl23, bat gsor23,
killifish MPIA_NFZ_2, tomato GCF_001406875.1, soybean JGI-PGF v2 2021,
barley and poplar unlabelled.
- Every position column exists in three generations (Mb, Mb_2016, legacy
mouse Mb_mm8; likewise Length/Length_2016/Length_mm8 on Chr_Length) and
the labels do not all match the numbers. Human is hg19 in both
generations (chr1 249,250,621; SOX2 chr3 181.43, not hg38's 181.71).
Mouse Length_2016 chr1 = 197,195,432 is mm9 exactly, while current
Length = 197,426,691 matches no standard build -- but mouse probes are
mm10-era (Sox2 chr3 34.5). Rat current Length chr1 = 282,763,074 is rn6
exactly, though MenuName advertises rn7. Geno.Mb_2016 exists only for
human, mouse, soybean, barley; the mouse-named Mb_mm8 is populated for
barley too.
docs/search-engines.md gains the coverage side: the index is a projection
of MariaDB with no species filter (only confidentiality < 1 AND public > 0),
so 14 species -- every Species row with public data -- with 44,842,506 gene
and 22,601 phenotype documents, dominated by mouse (27.7M), human (15.2M)
and rat (1.4M). The crawl reads the *current* columns, so the index inherits
hg19 / mm10-era / rn6-era coordinates, never a modern assembly; and rat
positions deserve suspicion (Sox2's 44 probes span chr2 1.2-117.6 Mb).
|
|
The ravanan path asks for the same variance components as the direct one:
gemma-loco.scm's gwa step passes -pve, so each chromosome's job writes
pve/se(pve)/vg/ve into its own .log.txt, and compute_ravanan turns them
into the same note (every chromosome, plus the extremes with vg/ve) and
the same unrounded "variance:" lines in the ?id= run log.
It reads them from the extracted GWA archive rather than from the copy
gemma-wrapper keeps beside it, and that is the point of doing it here:
the copy beside the archive is a 5eb53fa feature, while a ravanan job
runs whatever gemma-wrapper its own Guix manifest resolves, which need
not be that new. The .log.txt has always been inside the archive, and
the archive is unpacked anyway for the chromosome's .assoc.txt -- so
read_variance_components_from_logs labels each log by the "-loco <chr>"
in its command line, falling back to the <hash>.<chr>.assoc.txt.log.txt
file name, and sorts chr1..chrX.
Verification, precisely: the CWL compiles with -pve (ccwl compile, checked
by hand), the archive reader is unit-tested (labelling from the command
line, from the file name, numeric ordering, a log without the block
dropped), and a real archive from this deployment does carry the block.
An end-to-end ravanan request could NOT be completed on the dev box, for
reasons that have nothing to do with -pve and are now written down in
docs/ravanan-slurm.md: ravanan's own Guix evaluation of the manifest
fails ("rust-pyo3-0.26.0: unbound variable" -- the ambient channel set,
since the production process does not set GUIX_PACKAGE_PATH at all, so
this endpoint cannot resolve gemma-wrapper there), its intern of inputs
hard-links and so needs store and inputs on one filesystem ("Invalid
cross-device link"), and the one job that did start sat "Waiting for
lock" on gemma-wrapper's $HOME/<hash>.lck until the call timed out.
|
|
The note reported the range and named the two chromosomes that bounded it,
which is not enough to answer the question the numbers exist for. Under
--loco each pve belongs to a *different* kinship matrix, and GEMMA's
per-chromosome .log.txt files live on the server -- so a caller told "0.31
to 0.95" could not ask which chromosome was which, or compare a single
chromosome against BulkLMM.
The note now lists all twenty, compactly, with the range and the two
extremes keeping their vg/ve:
GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451,
chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to
0.9549 over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg
23.3659, ve 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933,
ve 0.8661).
and streaming::append_lines puts the unrounded values in the run log as
variance: chr7 pve=0.954938 se_pve=0.357032 vg=76.0933 ve=0.86612
so a ?id= run's /compute/gemma/status stream carries them while the run is
still in flight, rather than only in a reply someone has to keep. Both
measured on BXD::10001 through the dev server before committing.
The values are printed with four significant figures (scientific when the
magnitude needs it), unchanged: rounding variance components to four
decimals was the earlier bug, and these are the same numbers a reader
compares with BulkLMM's h2.
Docs: README's /compute/gemma section, SKILL's gemma bullet ("quote
per-chromosome values, not an average: they are twenty different kinship
matrices") and docs/bulklmm.md's example.
|
|
Once both engines' variance components were readable, BXD::10001 (34
animals) gives:
BulkLMM, global null h2 = 0.819 (sigma2_e 5.381)
GEMMA, LOCO per chr pve 0.3125-0.9558, median 0.8015, mean 0.7769
0.80 against 0.82. The engines never disagreed about the variance split;
they disagreed about which chromosome's kinship to hold the rest of the
genome out from. That closes the question this comparison started with.
Also says how to quote them: GEMMA's pve with its range because it is per
chromosome, BulkLMM's h2 as one number because it is global, and neither
set of p-values against the other.
|
|
GEMMA_WRAPPER_CMD is the lever: the profile's gemma-wrapper may be older
than the pin, and pointing the variable at the store path is enough --
-pve reaches GEMMA either way, but the per-run .log.txt files where pve
lives are only kept beside the archive from 5eb53fa on.
Recorded the verified production state too: both processes restarted with
gemma-wrapper-1.00-pre1-4.5eb53fa, and POST /compute/gemma for BXD::10001
on the running instance returns pve 0.3125 to 0.9558 over 20 chromosomes
-- measured there, not on a fixture.
|
|
The GWA step now passes -pve, and the reply gains a note with the
variance components GEMMA computed for the null model. Under --loco
there is one per chromosome, so the note gives their range and names
the two that bound it:
GEMMA -pve, null model, per chromosome (LOCO): pve 0.3118 to 0.9549
over 20 chromosomes -- lowest 8 pve 0.3118 +/- 0.5009 (vg 23.3659, ve
12.3389), highest 7 pve 0.9549 +/- 0.3570 (vg 76.0933, ve 0.8661).
That is BXD::10001 measured, not assumed. pve is the proportion of
variance the kinship explains -- the same quantity as /compute/bulklmm's
h2, per chromosome rather than genome-wide -- which is what the
16627/16769 comparison against BulkLMM needed.
Reading the numbers took three decisions worth stating:
- Four decimals is wrong for variance components. The first live run
reported 'vg 0.0000, ve 0.0000' for a BXD::16769 null fit whose real
components are 3.411e-5 and 3.411e-10 -- both rounded to zero, which
reads as 'no variance' when the split is 1.0. Components are printed
with four significant figures, scientific when the magnitude needs it,
the way GEMMA's own logs do with %g.
- se(pve) is -nan on a boundary, and Rust parses -nan as a NaN f64, so
the numeric parse alone let it through as NaN. Non-finite values are
now 'not available': a boundary estimate has no standard error.
- Twenty chromosomes of values bury the other notes, and every value is
in the .log.txt beside the archive anyway, so the note names the ends.
Requires gemma-wrapper 5eb53fa, which keeps each run's .log.txt beside
its archive instead of only inside the .tar.xz.
Docs: README's /compute/gemma section, SKILL's gemma bullet (compare pve
with h2, never vg with sigma2_e -- pve is scale-free), and
docs/bulklmm.md's 'what would settle it', which now says the plumbing is
in rather than pending.
|
|
Tested -lmm 9 with -pve on a synthetic BIMBAM trio (40 individuals, 60
markers, GEMMA 0.98.6): the flag is accepted and the log carries
## pve estimate in the null model = ...
## se(pve) in the null model = -nan
## vg estimate in the null model = 0
## ve estimate in the null model = 0
So what was missing is plumbing, not support. Two gemma-wrapper changes are
in its working tree (5eb53fa): the per-run logs are copied beside the
archive -- they were only inside the .tar.xz -- and a failing child's own
ERROR lines are printed before the Ruby backtrace. Once pushed and pinned,
/compute/gemma can pass -pve and report pve/vg/ve per chromosome beside
l_mle, which is what the 16627 and 16769 comparison needs.
Also recorded: on random data GEMMA returns se(pve) = -nan with vg = ve = 0,
so a boundary pve is a direction rather than a proportion -- the same
caveat as BulkLMM's h2, from the other engine.
|
|
The earlier note said passing -pve crashes gemma-wrapper. It does not --
my hand-made invocation fails identically with and without the flag, inside
the parallel batch of per-chromosome jobs, so the flag was never isolated
and its compatibility with -lmm 9 under --loco is unknown.
What that control run did find is the real defect: the wrapper does not
surface the failing job's error (the batch's stderr goes to a temporary
directory that is removed), which hides every GEMMA failure, not just this
one.
|
|
Tried it: passing -pve through gemma-wrapper after -- crashes the wrapper
(Ruby backtrace, no output), and the logs the wrapper keeps are the kinship
step's -- 223 .log.txt in the cache dir, none containing a pve block,
because the per-chromosome LMM logs live in the wrapper's temp dirs and are
not retained.
Two routes remain, both written down: run GEMMA ourselves (-gk for a text K
matrix, then -lmm 1 -pve on one marker, seconds each) and parse 'pve
estimate in the null model'; or get -pve into the wrapper's allowlist
upstream. The first needs the kinship as plain text, which the wrapper only
caches as JSON.
|
|
settle it
Written down from a real analysis (BXD.8, 35 phenotyped animals): trait
16769 has both engines saying kinship dominates (BulkLMM h2 = 1.000,
sigma2_e = 0; GEMMA clamps l_mle on every marker but the chr2 peak), while
16627 is a genuine disagreement (h2 = 0.000 against a clamped l_mle
everywhere). Both sit on a boundary, which is expected at n = 35 -- but
opposite boundaries mean the engines are estimating different things:
* different kinship matrices (GEMMA's own relatedness from the BIMBAM
dosages, one per chromosome under LOCO, against calcKinship's
2*(X-0.5)(X-0.5)'/p + 0.5 with a unit diagonal);
* different encodings (0/0.5/1 with NA, against the same dosages with
missing values mean-imputed, against R/qtl's two-of-three codes) --
and 16627 is nearly determined by the chr2 genotype;
* different variance-component step (once under the null against
per-chromosome LOCO, clamped differently at a boundary).
The three measurements that would settle it, with what is known about each:
1. GEMMA's pve: it prints "pve estimate" when run with -pve, and
gemma-wrapper keeps 223 per-chromosome .log.txt files under its cache
dir -- none contain pve, because the flag is not passed. One argument
on the GWA step plus a parser.
2. BulkLMM's profile over an h2 grid: BulkLMM.profile_LL(y::Matrix, G,
covar, K, h2_grid, markerID; reml) is internal (not exported) and
returns (ll_list_null, ll_list_alt); it wants a one-column phenotype
matrix and a covariate matrix. An attempt to call it from the driver
is not landed -- the signature mismatch is the whole of the remaining
work.
3. The kinship matrices, or a summary: calcKinship is one line and can be
written per panel, GEMMA's is already cached as *.k.json; correlating
off-diagonals says whether the engines disagree about relatedness
before arguing about variance components.
Until those exist, the reading in the analysis holds: BulkLMM's h2 and the
thresholds that rest on it are approximate on a boundary, and GEMMA's
16627 p-values come from a kinship-dominated model.
|
|
The GTEx problem -- it is human-only, and a mouse symbol there is an empty
report -- is answered by the endpoint that already knows which entity it
matched: /genes/{symbol}/aliases now returns
label the entity's canonical symbol in that species ("Shh" for mouse,
"SHH" for ?species=human) -- the symbol a human-only service needs
links [{name, href, description}] built for that symbol and species:
RGD, MGI, BioAlliance, NCBI Gene, Wikidata (the QID), UniProt,
STRING, GeneMANIA and GeneCup, plus GTEx/Open Targets/Protein
Atlas/GWAS Catalog for a human entity or BioGPS/Allen Brain Atlas
for a mouse one
So a mouse Shh reaches GTEx through the human ortholog's label rather than a
guessed one, and a client no longer holds the URL vocabulary. Only links
that answer are built: HomoloGene, UniGene, Genotation, GeneBridge and
PhenoGen are gone, and Panther is browser-only (see SKILL.md "Gene
link-outs", where each URL was checked).
A cached entity written before the label was kept is refreshed rather than
served -- otherwise the link-outs would be built on a symbol it does not
have -- and `stale: true` still says when Wikidata was unreachable and the
cached answer was used.
tests/wikidata.rs: a mouse gene gets the mouse services and no GTEx; the
human ortholog gets label SHH and a GTEx href built from it. Both halves
skip when the reply is stale, since a Wikidata timeout is not a failure.
|
|
GeneNetwork2's webqtlConfig.py holds the link-out URLs its trait page shows
by gene symbol; twelve of them were still alive when checked (2026-11-11,
HTTP 200 and, where the page is not a JavaScript shell, the gene found), and
the SKILL now carries them as a table with descriptions and the URL
templates to hand a user:
RGD (gene search, mouse/rat/human -- the best single hub for both rodents),
MGI, Alliance of Genome Resources, NCBI Gene, Wikidata (the QID
/genes/{symbol}/aliases already returns), UniProt (symbol search or
accession), Ensembl (gene id), GTEx (human symbols only), STRING,
GeneMANIA (per species), Open Targets, Human Protein Atlas, GWBI GWAS
Catalog, BioGPS, Allen Brain Atlas, GeneCup.
Named so they are not offered by mistake: HomoloGene (NCBI retired it 2021),
UniGene (410), Genotation and GeneBridge (dead), PhenoGen (404), chibi
GEMMA, wiki-Pi; PANTHER answers 403 to a bot but works in a browser.
Two conventions: quote the species the symbol came from, and pick the
species-appropriate service (a mouse symbol in GTEx is an empty report).
The API hook is noted but not built: a `links` array on
/genes/{symbol}/aliases, so a client need not hold the URL vocabulary.
|
|
Checked what the plot actually draws: it has never drawn the alpha 0.10
line -- the plotter matches 0.05 and 0.01 only. What lists all three is the
reply's notes ("alpha 0.01: LOD 4.58, alpha 0.05: 3.65, alpha 0.1: 3.50"),
which is where the 0.10 came from.
Still, the point stands on its own: 0.10 is the loosest bar in the set and
drawing it invites reading it as the line to clear. So the defaults are
stated and pinned -- 0.05 and 0.01 on the plot, 0.10 only on request
(?all_thresholds=true, or an explicit ?thresholds=) -- alongside GEMMA's
Bonferroni, which is opt-in for the same reason: a reference line is an
annotation, not a result.
tests/bulklmm.rs counts the lines: exactly two by default, and no
alpha 0.10 label.
|
|
The OpenAPI description still promised "a Bonferroni reference line by
default", and GEMMA's own suggested plot call said the same. Both now say
what the code does, and where the number comes from: the plotter computes
0.05 over the analysed markers -- conservative under linkage and unrelated
to relatedness -- and ?thresholds= takes one computed elsewhere, each line
labelled with its source.
Also corrected in the README's /plot section and in SKILL.md, where the
same promise had been copied.
|
|
The marker blocks a single-trait scan is split into were the driver's
default (Threads.nthreads()) and changed with --threads. They are a
separate knob now:
BULKLMM_THREADS Julia threads (--threads) default 4
BULKLMM_BLOCKS independent marker blocks, scanned default 4
in parallel (1 = serial scan)
Blocks are exact either way -- LOD 9.589938 and the same permutation
thresholds at 1 and 4 -- so the knob only buys wall clock. The reply says
what ran ("split into 4 independent blocks and scanned on 4 Julia
thread(s) (BULKLMM_BLOCKS / BULKLMM_THREADS ...)"), because the answer to
"is it multicore?" should be in the response and not in the log.
|
|
We do not run GEMMA permutations -- each repeats the whole LOCO scan and the
gate gives GEMMA one slot -- but "not supported here" is not the same as
"not available". The refusal now says both:
1. POST /compute/bulklmm with nperm: 1000 -- an LMM whose thresholds come
from rotations that keep the kinship (R/qtl's nperm does not, and is
anti-conservative on a structured panel);
2. run GEMMA's own permutation locally. The reply's notes already name the
bimbam pair and the phenotype file, so `gemma-wrapper --permutate` or
`gemma -lmm 9 --permutate` on those inputs reproduces exactly the null
GEMMA would have used -- hours instead of seconds, which is the honest
price of that particular null.
docs/bulklmm.md gains the table this belongs in: three ways to get a
threshold, ordered by how honest the null is for a structured panel
(rotation with kinship; GEMMA's own LOCO permutation run locally; R/qtl's
nperm as a sanity check only). SKILL.md says the same in two lines.
tests/gemma.rs now asserts both halves of the refusal, and the earlier
assertion on /compute/rqtl -- which stopped being true when the refusal was
rewritten -- is replaced rather than left passing by accident.
|
|
docs/bulklmm.md records the three things that are easy to get wrong when a
BulkLMM result is quoted:
* where it stands against the other engines -- Haley-Knott with no
kinship (R/qtl), LOCO LMM whose permutations are refused (GEMMA),
LOCO LMM on HMM probabilities (R/qtl2), and this: a global-kinship
LMM whose permutation thresholds are rotation-based and therefore
cheap -- the only engine here that can say "genome-wide significant
at alpha 0.05" against a null that respects relatedness;
* what it is not: global kinship rather than LOCO, so its p-values are
not GEMMA's; h2_null says how much relatedness it found (near zero
here, which is why it reproduces R/qtl's 9.59 where GEMMA differs);
one degree of freedom, so multiparent scans stay with /compute/qtl2;
and no standard errors in 1.2.2's scan(), so the rows are
marker, chr, pos, lod, beta, h2;
* the cores, measured: Threads.@threads is in bulkscan.jl and
bulkscan_helpers.jl, never in scan.jl, so a single-trait scan is
serial by design and the visible parallelism was BLAS (96 threads
10.5 s against 4 threads 7.7 s). What the driver does about it --
exact marker blocks, cached matrix and kinship, capped BLAS, and the
breakdown printed into the reply -- with the numbers (20.9 s at one
block against 18.3 s at four, same LOD 9.589938).
Also the packaging picture, since it is not obvious from the outside:
guix-bioinformatics has a working julia-bin and a julia-build-system, but
only three of BulkLMM's closure are packaged, so a pure-Guix build is
35-45 recipes; the pragmatic route is julia-bin plus a depot built once
from the pinned Manifest. scripts/prepare-julia-depot.sh is that step,
written down, and the depot stays small and GPU-free (79 packages,
139 MB) with JULIA_PKG_OFFLINE=true at request time.
The plot test learns that both thresholds are drawn, not one.
|
|
Reported: a BulkLMM run does not look multicore. It is not a
misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint
calls -- has no `Threads.@threads` at all; the package threads
`bulkscan()` (many traits) and bulkscan_helpers, not scan. The
parallelism that was visible before was BLAS (OpenBLAS defaulting to all
96 cores here), which is why `user` time ran 4x `real` while Julia
threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s.
* The driver splits the markers into independent blocks and scans them
on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact:
the LOD for each marker is independent given the null variance
components, which depend on y and K only. Verified identical -- peak
LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks --
and permutation thresholds combine by taking each permutation's
maximum across blocks before the quantiles, which is what a
genome-wide threshold is.
* The parsed matrix and the kinship are cached per panel (read 1.3 s ->
0.4 s, kinship 0.9 s -> 0), because both depend only on the cross.
* The summary in `notes` now says what ran: julia threads, blocks, BLAS
threads, and the read/scan split.
* The hand parser is back: CSV.jl into a DataFrame is an order of
magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s).
* BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for
one scan).
Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks
18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of
the run and each block re-estimates its variance components. Where the
cores pay is bulkscan() -- many traits in one process, threaded over
trait blocks -- which is the shape the next endpoint wants; documented in
the README rather than left as folklore.
|
|
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values
have no threshold we can compute: a GEMMA permutation repeats the whole
scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual
stand-in and it is the wrong null -- it shuffles phenotypes as if the
individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global
kinship matrix and permutes by rotating the data through the kinship's
eigen decomposition, so a threshold costs seconds.
POST /compute/bulklmm {group, genotype_version, samples|basket,
nperm, seed, reml, assumption}
Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8,
35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2
4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81,
0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's
published locus and LOD, and a significant peak against a threshold this
API could not compute before.
Two differences from GEMMA are in every reply, not implied: a global
kinship matrix rather than leave-one-chromosome-out, and h2_null saying
how much relatedness was found (here ~0, so the fit is effectively
fixed-effects -- which is why it agrees with R/qtl where GEMMA differs).
* src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a
strains x markers matrix (our BIMBAM pair is already the allele
dosages BulkLMM wants), a marker map, the phenotype and covariates
-- runs the driver through streaming::run_streaming_with_env, and
reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid
f64 in Rust and would otherwise be reported as a value.
* scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds),
with the seed recorded so a threshold set is reproducible.
* gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is
cheap, unlike R/qtl's.
* Julia comes from the deployment's own store binary with a prepared
depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request
runs JULIA_PKG_OFFLINE=true, so a scan can never download anything.
`--project=<path>`, not `--project <path>`: Julia 1.8 takes the
space-separated form as a program argument.
tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and
seed are in the notes, and a scan without permutations says there is no
threshold rather than leaving a bare LOD to be read as significant.
|
|
Two engines reading one riset panel can fit different predictors, and the
reply now says which one it used and what it cost -- with this panel's own
numbers.
* `genotype_model: "riself" | "f2"` on /compute/qtl2 (`het` kept as the
older spelling: drop/keep). riself is the default: R/qtl's
convert2riself coding, the H calls become missing and the HMM
re-imputes them. f2 keeps them as a third genotype -- a diagnostic.
* The conversion now counts the calls it drops and keeps the count
beside the cross (conversion.json), so a reply can say
"610684 heterozygous call(s) (12.7 %) were dropped". A cross
converted before that file existed measures the count once, on first
use, rather than never.
* Crossed with the note that already explained the H drop, so the two
read together: what riself does, how many calls it cost, and what f2
would give.
Documented in the README as "Why R/qtl and R/qtl2 disagree on a riset
panel", with the measurements (BXD trait 16627, BXD.8: R/qtl 9.59 at
rs33299937; qtl2 riself 3.73; qtl2 f2 9.68), R/qtl's adaptive
convert2riself rule quoted from its own source, and qtl2's side: riself
has two states, f2 three, and `genril2` -- from qtl2 0.18, "the progeny
are inbred lines", cross_info first column the number of generations --
is the honest model for residual heterozygosity and is available in the
0.42 we run. What is missing for it is recorded precisely: a founder-based
cross expects individual genotypes coded as founder alleles rather than
1/2/3 (a prototype with numeric codes read every genotype as missing), so
founder_geno from the two parental alleles and an allele-coded het remain
to be done, with DOL as the counter-example (its H is a collapsed DO, not
residual heterozygosity).
tests/qtl2.rs: the model is reported (Cross: riself / Cross: f2), the
dropped count is in the reply, keeping the calls finds more, and an
unknown model is refused rather than ignored.
|
|
Returning genotypes is the one thing this API otherwise never does, so it
is fenced and the fence is the feature:
GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker>
* the server re-runs the R/qtl scan for the trait -- the same machinery
and gate as /compute/rqtl -- and takes the peak from that run, rather
than trusting a client that names a marker;
* the marker must lie within ?window= Mb of that peak (default 10), or
the answer is 422 not_the_peak quoting the peak's marker, chromosome,
Mb and LOD;
* the peak must reach ?min_lod= (default 3), or 422 not_a_hit;
* only animals that have a value for the trait are listed, and only
this marker's calls. Nothing else of the matrix leaves.
On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals
across A 110 / B 8 / H 74 -- the two a class of eight being exactly the
thing worth seeing.
POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's
`animals` (or its `genotypes` map) with the trait's values and get an SVG
with one point per animal, genotype class across, value up, class means
marked. Nothing is recomputed -- the plot is of the numbers the client
sends, like the other plots. New script scripts/plot-genotypes.py
(stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key.
tests/peakgenotypes.rs: the peak's calls for animals with values (with the
counts agreeing with the rows), the SVG, and a marker on another
chromosome refused with the peak quoted.
Also keeps /compute/rqtl's error order: naming both a group and a file is
"ambiguous" even when the request also forgot its values.
|
|
Two things asked for, and one bug each.
* POST /uploads/genotypes with an existing name now REPLACES it. A
genotype is the user's file to keep, not a record the deployment
curates; the digest in every reply says which version ran.
* genotype_upload on all four engines -- /compute/{rqtl,gemma,qtl2,
fitqtl} -- in place of genotype_file/group. The server writes the
form each engine reads: a GN .geno for R/qtl and R/qtl2, a BIMBAM
pair (new: genoupload::to_bimbam, coding from the base's
@mat/@het/@pat, NA for missing, snps as marker/bp/chr) for GEMMA. No
user path travels with a request.
Verified against the full 3811-marker BXD.4 matrix uploaded as an update:
rqtl 3811 rows, gemma 3809, qtl2 3730, fitqtl by marker.
The bug worth naming: values given by name were being aligned against a
file, and for GEMMA that file is BIMBAM, which has no individual names --
so the phenotype file came out empty and the wrapper failed with a Ruby
backtrace that said nothing. GEMMA's values are now ordered by the
upload's own individual list before the pair is written. The related
fixes: rqtl and fitqtl align named values onto the file's individuals
(they previously demanded its column order), and genotypes::align_to_file
is the one place that does it.
tests/genoupload.rs: mapping an upload on rqtl/qtl2/fitqtl with values by
name, and replacing a name (one row in the listing, a new digest). GEMMA's
LOCO kinship needs more markers per chromosome than a small fixture has,
so it is exercised against a full panel rather than in the fixture.
|
|
Three follow-ups from putting an upload through the whole loop.
* /compute/qtl2's `genotype_file` branch only accepted positional
`trait_values`, i.e. it demanded the file's column order -- which a
caller with an uploaded population does not have. It now takes
`samples` (name and value) or a `basket` and aligns them onto the
cross by name, leaving individuals without a value as missing.
Positional `trait_values` still work for a file whose order is known.
* DELETE /uploads/genotypes/{name} now removes the .geno it wrote on
demand: a file nobody can name is a file that confuses the next
operator.
* scripts/example-genotype-upload.py spreads its markers across
chromosomes. One chromosome of markers is a poor example -- a
leave-one-chromosome-out kinship has nothing to leave out, so the
scan came back with no rows at all, which looked like a broken upload
and was really a property of the example. 20 markers over the genome
map cleanly (19 rows, the 20th used for the HMM), and the README says
to use "kinship": "overall" if a file really has one chromosome.
|
|
Tab-delimited, header names the individuals from column 4, then marker,
chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own
SQLite file -- nothing goes near MariaDB -- and written out on demand as
a GN .geno carrying the base's @-metadata, so the R scripts read exactly
what was uploaded.
An upload updates a population, it does not invent one, and the checks
say so:
* the base version must exist (resolved through the same
genotypes::resolve_named the rest of the API uses);
* at most 50 individuals the base does not have, and at least one that
it does -- no overlap is a different population, not an update;
* at most 200 000 markers, and the refusal says the sysadmin has to
organize a file that size, because this is a request path, not an
ingest;
* the values must be the same kind as the base's: letters, whole
numbers or probabilities. A letters panel holding 0/1 would be read
as the wrong thing silently, so it is refused naming both kinds.
GET /uploads/genotypes lists them, /{name} its metadata (overlap, new
individuals, kind, digest), /{name}/markers pages the marker table, and
/{name}/geno writes the file and returns the path for `genotype_file` on
the compute endpoints. DELETE removes one.
scripts/example-genotype-upload.py turns any .geno into a valid update --
a subset of markers, a few individuals dropped and added -- and
tests/genoupload.rs builds its own from the deployment's BXD.4 for the
same reason: upload, page the markers, write the .geno and check the
layout, delete, then break each rule (no overlap, numbers in a letters
panel, commas) and check the refusal says which.
Unit tests cover the three value kinds, missing cells, ragged rows, a
non-numeric position, the 50-new-individual limit, and the .geno layout
(Chr first, then the marker -- GN's order, not the upload's).
|
|
The other half of changing data: bringing new columns in. A matrix is a
delimited table -- a header row, then one row per individual, the first
column naming them -- stored in the uploader's own SQLite file and checked
against an existing genotype version's panel before it is accepted.
* POST /matrices {name, genotype_version, contents, kinds?}: parse the
table (tab, comma or semicolon, guessed from the header), then check
it against that version's individuals. The count must match and every
name must be in the panel; 92 of 93, or a name the panel does not
have, is a 422 that says which. A mapping that silently drops
individuals is worse than an error.
* kinds marks columns that are covariates; everything else is a trait.
* Each column becomes a basket named <matrix>.<column> -- a dot, not a
slash, because a basket name has to survive a URL path segment -- so
a column maps on any engine with {"basket": "my-traits.bw"} and
inherits the revision history and digest without a line of new
mapping code.
* GET /matrices lists them, GET /matrices/{name} their columns and
basket names, GET /matrices/{name}/values the table back (parsed
values, null where missing), DELETE removes the matrix and its
columns.
Two bugs this found in the basket store, both real:
* SQLite has foreign keys OFF by default, so the ON DELETE CASCADE
never fired: deleting a basket left its rows behind, rowids were
reused, and the next create hit "UNIQUE constraint failed:
basket_version.basket_id, revision". The pragma is now set, and
deletion removes the dependants explicitly, because files written
before it have orphans.
* The first attempt named a column's basket "<matrix>/<column>", which
the basket-name rule (rightly) refuses. The dot keeps both rules.
scripts/example-matrix.py writes a table from a genotype file's own header
with deterministic values, so the fixture passes the panel check by
construction -- tests/matrices.rs builds its own for the same reason and
walks the loop: upload, read back, map a column with R/qtl2, delete, plus
the three refusals (too few individuals, a name not in the panel, an
unknown version).
|
|
Three things the first cut was missing, and both were asked for.
* An automatic version history. Revision 1 is the column as first
stored and each PATCH adds one -- with its own digest and Unix
timestamp -- and earlier revisions are kept. GET
/baskets/{name}/revisions is the history, ?revision=N reads an old
column, and a mapping can pin one: {"basket": "wbc-edit@2"}. That is
the point of a digest: a result quoted yesterday can still be
reproduced after the column has moved on.
* Files named for the user, not a hash of the subject:
BASKET_DIR/<user>.sqlite (0600, WAL). Tokens rotate, subjects do not,
so a new token finds the same baskets -- and an operator can see
whose baskets a file holds. A subject that is not filename-safe is
sanitised and disambiguated with a short digest, and a pre-naming
hash file is renamed on first use rather than orphaned.
* /compute/rqtl and /compute/gemma and /compute/fitqtl take `basket`
too, so all four engines share one meaning: a basket resolves to the
same (name, value) pairs a client would have posted, and every reply
records the revision and digest it ran.
The schema upgrade is a real one, not a reset: an older file's single
column becomes revision 1 of the history, cells copied across with the
digest and timestamp they already had. Tested by building the old schema
by hand and opening it (an_older_file_is_upgraded_with_its_column_intact),
which is also how I found that the migration was recreating a table the
base schema had already made.
Verified on all four engines, head and pinned: rqtl 3811 rows, gemma
3809, qtl2 3730, fitqtl 3 markers -- each reply carrying "Basket '...'
revision N (digest ...)".
|
|
The next phase is changing data, and the constraint is that MariaDB is
never written to. A basket is the storage half of that: take a trait's
values, edit some of them (or build a covariate column that does not
exist), keep the column under a name, and map that instead.
* POST /baskets stores a complete column; GET /baskets lists yours;
GET /baskets/{name} its metadata; GET /baskets/{name}/values the
cells; PATCH /baskets/{name} replaces the column with a new revision;
DELETE removes it.
* POST /compute/qtl2 takes "basket": "<name>" in place of `samples`,
and both at once is a 422 rather than a silent choice. The mapping
code below it is untouched -- a basket resolves to the same
(name, value) pairs a client would have posted.
* One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600,
WAL, busy_timeout. `GET /baskets` only ever lists the caller's own;
the file name is a hash, so a directory listing does not name users.
* The full column is stored, not a diff -- the caller has the original
beside it, and a mapping needs one complete column.
* Text with a parsed copy beside it: raw (what was typed), value (the
number or NULL), status (ok/missing/bad). "NA" and 0 stay different
claims, and a column with something that is neither a number nor a
missing marker (NA, x, -, ., ?) is refused with the samples named --
silently dropping it would change a trait without saying so.
* A revision and a digest: an edit bumps the revision and changes the
digest, and a mapping reply records both ("Basket 'wbc-edit'
revision 2 (digest 9559a70e...)"), so two runs on one name cannot
look alike. A result cache must key on the digest, not the name.
The loop, end to end on BXD trait 10001: 34 values posted as a basket,
one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2
-- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the
revision and digest.
tests/baskets.rs walks that loop (create, list, values, patch, map by
name, delete) and the refusals: a non-number named in the error, a name
that cannot become a path, an unknown basket, and basket-plus-samples.
Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest
changing on edit, per-user isolation, and name validation.
rqtl/gemma take the same 10-line change next -- their `samples` shapes
differ, so they are a separate commit rather than a rushed one.
|