| Age | Commit message (Collapse) | Author |
|
stale tests
`ccwl compile` emits CWL as YAML (its only other target is dot), so the
new input check must read that shape -- it was parsing JSON and would
have turned a working request into a 502. A hand-written scan of the
`inputs:` block, deliberately not a YAML dependency: ccwl's output is
machine-generated and unvaried, and adding serde_yaml would mean
regenerating the crate list in guix.scm for one small check.
Two tests were stale in ways only the package's own check phase shows:
- tests/auth.rs built AuthToken literals without the new `scheme`/`id`
fields, so the crate's integration tests did not compile;
- test/python/test_envelope.py asserted that /version carries no `next`
links, which stopped being true when every read endpoint gained them
("always offer the next step"). The deployed build fails that test
too, so it is not a regression from this work -- but it did block
`guix build -f guix.scm` at the python-rest-test phase.
Verified with the packaged binary (guix build -f guix.scm, all phases
including both test suites): POST /compute/gemma/ravanan through ravanan
and the Slurm container returned 200 with 636 rows in 13.6 s, on a
request authenticated with a gn4ai_ short token -- the log line naming
user zachs, scheme opaque and token smoke-2.
|
|
A second credential format beside gn-auth's JWT, for agents and scripts:
`gn4ai_` + 56 body characters + 2 checksum characters = 64, from an
alphabet with no look-alikes, so a typo is refused before the registry is
consulted. Opaque by design -- nothing is stored in the token; every fact
lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode
0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row,
takes effect on the next request, and a token in a log line reveals
nothing; lose the registry and every short token stops working.
Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and
GET /tokens/who are behind a router layer that refuses any caller whose
TCP peer is not 127.x.x.x/::1, and any request claiming to come through a
proxy. A layer rather than a per-handler check, so nothing of a remote
request is parsed before it is refused; there is no configuration switch,
because minting credentials must never be reachable from the network.
A short token reads public data only: every token-only endpoint, and
`#[gnauth(secured)]` endpoints for public rows; a confidential dataset is
refused with a 403 that names the JWT route, since gn-auth cannot validate
an opaque token (the on-the-fly-bridge and registry-grant alternatives
were both rejected for v1).
scripts/gn4-token.py create|list|revoke|who drives the routes, so there is
one implementation of the format and the revocation rules; the token is
printed exactly once.
Tests: the localhost guard over all four routes (remote peer, no peer,
proxied header), and mint -> authenticate -> who -> revoke -> 401.
|
|
Two follow-ups to the parental-design endpoint.
/groups/{group} now carries the headline in its notes -- "Parental design:
C57BL/6J (maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes
the parents and F1s by default" -- plus a next link to the full design. "Which
cross is this?" is two strain names, and needing a second request to see them
was the only reason the endpoint had to be called first. A group with no
metadata says so in the same note, pointing at parents_and_f1s.json and at
`individuals.exclude`, rather than leaving the reader to wonder why nothing was
dropped.
The metadata is the same fact in two places (GN2's ParInfo, our JSON) and it
drifted once already, silently: the paternal/maternal labels were swapped
against GN2's unpacking, and nothing noticed because the exclusion rule takes
both strains regardless of role. scripts/check-parental-metadata.py parses
ParInfo with ast (no gn2 import) and compares it group by group, reporting
disagreements per role and treating groups the JSON knows but GN2 does not as
informational. tests/parental_metadata.rs runs it while a GN2 checkout is
present (GN2_CHECKOUT, or a sibling path) and skips once GN2 is retired -- at
which point the JSON is the only copy and there is nothing to drift from. It
reports 23 groups reproduced, 1 extra (BXD-Longevity).
|
|
GET /groups/{group}/parents-and-f1s answers the question the mapping notes only
implied. It reports the F1 strains, the maternal and paternal strain (from the
deployment's parents_and_f1s.json, ported from GN2's ParInfo -- all 24 groups),
the @type of the group's genotype file, the individuals each role names where
the metadata names them (BXD-Longevity does, most do not), and -- the part a
caller needs -- which members of the current panel the default rule drops and
which the `F1` name heuristic catches.
Two things that came out of building it and comparison with GN2:
* the JSON's paternal/maternal labels were INVERTED against GN2's reading.
GN2 unpacks ParInfo as `f1, f12, maternal, paternal`, so BXD's maternal
strain is C57BL/6J; our file said paternal. No mapping was affected
(non_strains takes both regardless of role) but the metadata was wrong, and
anything using "maternal" later -- reciprocal F1 direction, the @mat/@pat
note -- would have been backwards. The file is rewritten from ParInfo itself
(ast-parsed, not retyped), 23 groups' roles filled in and the 24th
(BXD-Longevity, which GN2 does not have) left as it was.
* an RI set has two kinds of F1 name: the design's (B6D2F1) and the panel's
own intercross individuals (C57BL/6JxBXD037F1). The endpoint reports
excluded_from_panel and excluded_by_name separately and says which is doing
the work, because "0 declared, 37 by name" is otherwise confusing. Empty F1
names (C57BL-6JxC57BL-6NJF2 has two) are dropped rather than becoming
exclusion entries that match nothing.
genotypes::non_strains is rebuilt on the same typed parser (DesignRole /
ParentalDesign), so the rule and the endpoint cannot disagree. tests: a unit
test for the roles, empty-name handling and the unlisted-group source, and an
integration test asserting BXD's maternal/paternal orientation and the note.
|
|
Parents and F1s are excluded by default (as in GeneNetwork2/3), with
include_parents_f1 to keep them all. That boolean was not enough for what was
asked -- "include these, drop that substrain" -- and the exclusion list itself
is data (parents_and_f1s.json, seven groups) that no request can reach. F1
names also differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH,
C57BL/6JxBXD037F1 for the intercross F1s), so pattern matching is the wrong
tool.
Every group-mapping endpoint -- /compute/gemma, /compute/gemma/ravanan,
/compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm -- now takes
"individuals": {"include": [...], "exclude": [...]}
applied after the parent/F1 rule, exclude winning over include. An unknown name
is a 422 invalid_individuals whose message lists the panel: a silently dropped
animal changes a kinship matrix without anyone noticing, so the failure has to
be loud. genotypes::select_panel is the one implementation, and it returns what
it did (dropped parents/F1s, re-included by name, excluded by name) so the
Genotype note can say it: "197 individuals, 38 parents/F1s dropped; 1 excluded
by name".
The note now also spells out the distinction that makes this option
treacherous: the kinship uses *every* individual in the genotype file, the
association uses only those with a trait value -- GEMMA reads no phenotypes for
the kinship. So "include the F1s" changes the relatedness structure without
putting the F1s in the fit, and a request can legitimately have 234 individuals
in the genotype file and 34 in the association. Verified on BXD.8: excluding
BXD1 gives 197 individuals / 33 values; include_parents_f1 with an F1
re-included and a substrain excluded gives 234 / 34.
A basket supplies values by name, so a selection applies to a basket-fed
mapping the same way: the individuals excluded are simply not in the panel any
more, and their basket values are ignored.
GenotypeError gains Invalid for this (422) so a caller's typo is not reported
as a missing genotype file; the read-only endpoints that can never see one map
it the same way for exhaustiveness.
|
|
Two additions that make the endpoint usable rather than merely correct.
POST /compute/bnw/features -- from a network to a mappable trait. Every node
with at least one parent at the threshold is a module (the node plus its
parents), scored per sample by the first principal component of the module's
standardised members, sign-corrected so the score rises with the node. Without
?module= the reply is one row per module -- module, members, size,
variance_explained -- where variance_explained says whether the score is a fair
summary at all. With ?module=NAME it is that module's per-sample scores in the
{name, value} shape every mapping endpoint already takes, so the chain runs
network -> module -> /compute/rqtl or /compute/gemma with no translation layer.
That is what BNW is for in GeneNetwork; the diagram is a by-product. The notes
name the construction (a citable module score must say what it is) and repeat
the caveat: a module is a correlation structure, not a QTL.
GET /compute/bnw/examples and /compute/bnw/examples/{name} -- BNW's own worked
examples, for an agent to explore instead of inventing data. The five datasets
its tutorial and examples pages present are listed with their sizes, and each is
served as a matrix in the values shape /compute/bnw takes, with a ready-to-post
action carrying the body. ?columns= restricts the matrix (the spleen example
has 14 variables and a caller may want fewer). genotype_column warns that a
column is a genotype code: this driver is continuous-only, so it enters the
network as a number, the way the web application's GeneNetwork path treats it.
Files come from BNW_EXAMPLES_DIR (a gn-bnw checkout's sourcecodes/
example_datasets); a deployment without it answers 404 no_examples.
Measured on the dev instance: features over five BXD traits gives two modules
(BXD::10001 with BXD::10003 and BXD::10041, 60.4% of variance; BXD::10044 with
BXD::10001, 27.2%), and BXD::10001's 19 scores posted straight to /compute/rqtl
and /compute/gemma both return 200 -- gemma peaks on chromosome 6. tests/bnw.rs
covers the module list, one module's scores in mapping shape, the unknown-module
422, the example list, one example's matrix and its ready-to-post action, and a
404 for a name that does not exist.
|
|
Post the reply from /compute/bnw back and get an SVG: the same plumbing as
/plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/
svg+xml, the plotter's line in x-plot-summary), nothing recomputed.
Two figures, because one cannot say what the other can:
* the network (default): nodes on a circle, one arrow per edge that survived
the threshold, arrow width and label = the edge's posterior, and the
below-threshold edges as faint dotted lines -- so a reader sees what was
*considered* as well as what survived, which is half of why the posterior
table exists;
* ?kind=posterior: every candidate edge as a cell in a matrix, dark = high
posterior, outlined cells are the ones above the threshold. The diagram
cannot show that an edge came within 0.02 of the threshold; this can.
Pure-Python SVG, standard library only: no graphviz in the deployment. A
circular layout is less pretty than a force-directed one and is deterministic,
which matters more for a figure that accompanies a JSON result. Node labels
keep the distinguishing tail of a long id, with the full name beside the node.
Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw
reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the
strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the
second figure. tests/plot.rs covers both figures plus the 422s (not a BNW
result, unknown kind).
|
|
POST /compute/bnw learns a network over a set of variables and returns one row
per candidate edge: from, to, posterior, in_structure. Inputs are published
traits of a group (group + traits) or an explicit matrix (values: one entry per
variable with per-sample values), which covers the case the database cannot --
a basket's columns, a handful of probe sets.
BNW's knowledge stays in the gn-bnw repository. GN4 writes a variables x
samples matrix, runs `bnw learn` (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR) and
parses the reply: nothing about the three C programs, the private work
directory or the type line in the input file lives here, so BNW's pipeline can
change without a GN4 release. The driver's failures map onto the API's own:
bad input 422, a missing driver 502 bnw_not_installed (its exit 77), a failed
computation 502 bnw_failed in the driver's words.
Decisions worth recording:
* parameters are BNW's own defaults -- maxparents 4, k 1, tiers 3, threshold
0.5, read from its create_tiers_gom_part1.php rather than chosen here --
and the reply names the values used, because a network at 0.5 and one at
0.9 are different answers to the same question;
* at most 40 variables and at least 3 samples complete across every variable.
The first is the structure search's exponential in-degree; the second is
BNW modelling a complete multivariate normal, so a sample missing any value
is dropped -- and the note says how many survived rather than silently
shrinking the panel;
* its own gate, always claimed (threshold 0): there is no such thing as a
small BNW run, the cost is the search, not the number of individuals;
* the notes say plainly that a posterior is not a p-value and that a network
is not a QTL, because that is the easiest thing to over-read.
tests/bnw.rs compares the endpoint against the answer the BNW *web
application* gives for its own six-trait example (the same fixture as the
driver's parity check, gn-bnw/test/golden/): the five edges must match, and the
notes must name the parameters and carry the caveat. It skips when the driver
or its binaries are absent.
Not deployed to production.
|
|
cannot do
Two gaps, both invisible until a real request ran against a cluster.
The endpoint read only genotype_file, snps_file and trait_values from the
request; group, genotype_version and samples were accepted by the shared
request type and silently dropped. A /compute/gemma-shaped request
(group + samples) therefore arrived as empty paths and empty values and
died with "I/O error: No such file or directory (os error 2)" from
reading "". Running it exposed the second gap in the same breath: the
gate sized the request from trait_values.len(), so a samples-only request
was sized 0 and took no concurrency slot at all.
resolve_inputs is now one function in routes/gemma.rs, called by both
handlers: upload materialised into the pair the engine reads, basket
resolved to named values, group prepared with the parent/F1 and
empty-genotype rules, named samples aligned by name onto whatever
individuals the file has, positional values checked against the panel
size. That block was ~250 lines of duplicated rules inside compute_gemma,
which is how it drifted in the first place; the direct runner now calls
the same function and keeps only its covariates, normalization and gate.
The ravanan reply also carries the input notes now (which genotype
version, how many values aligned, what was dropped), so a run says what it
mapped as well as where the peak is.
Refused rather than ignored, because the CWL workflow has no step for
either: covariates (422 covariates_not_supported) and normalize (422
normalization_not_supported), each naming /compute/gemma as the endpoint
that can do it. A new test asserts both refusals happen before any job is
submitted, so it needs no ravanan, ccwl or cluster.
Verified against the Slurm container from gn-machines: group: BXD became
40 Slurm jobs (20 kinship + 20 GWA, scattered over the snps file's
chromosomes), all completed, 21 055 rows in ~3.5 min. README and
docs/ravanan-slurm.md record the four requirements that run turned up,
three of them not obvious: ccwl on PATH, a ravanan build with the
--slurm-api-endpoint fix, the *server's* HOME writable by the job user
inside the cluster (gemma-wrapper locks $HOME/.<hash>.lck there), and
GUIX_PACKAGE_PATH for the manifest.
|
|
A failing scan came back as
BulkLMM failed: in expression starting at scripts/bulklmm_scan.jl:26
which names neither the package nor the reason. Julia had said it plainly,
two lines up:
ERROR: LoadError: ArgumentError: Package CSV not found in current path.
- Run `import Pkg; Pkg.add("CSV")` to install the CSV package.
The complaint finder looked for a line starting "Error" -- Julia writes
"ERROR:" -- and fell back to the last non-empty line, which is the line
number. That is the same defect as gemma-wrapper's Ruby backtrace, in our
own error handling: the wrapped tool said why, and we threw it away.
julia_complaint takes the first ERROR:/LoadError line plus Julia's hint
line when there is one, with the last line kept only as a fallback for a
driver that fails silently. Three unit tests cover the three shapes.
The integration tests also stop failing on a half-instantiated Julia
project: bulklmm_available used to accept any directory with a
Project.toml, and /tmp/gj-env has one naming BulkLMM and nothing else, so
the scan died on `using CSV` instead of skipping. It now requires the
project to name the three packages the driver loads, which is what a
usable env is.
|
|
tests/envelope.rs follows every href in every reply, and running it end to
end found six that were not addresses:
- /species, /species/{name}/datasets and /groups/{group}/datasets offered
/species/{species} and /datasets/{dataset} -- templates, which 404 when
followed. Each now points at a real row from the listing in hand, and
the description says any other name can go in its place.
- /datasets/{name} offered /datasets/{dataset}/probesets/{probeset}: the
listing it already links is the callable one, so the shape moved into
that link's description.
- /traits/{dataset}/{trait}/values offered a GET
/traits/{ds}/{trait}/correlation, which does not exist: correlation is
POST /compute/correlation and that reply already carries it in
"actions". Removed, with the reason written where it was.
- the same reply built its /groups/... links from the *dataset* name.
BXDPublish's traits live in group BXD, and /groups/BXDPublish/phewas is
a 404 (no genotype files for it). db::get_group_name resolves the group
from InbredSet.Id and the links use it; when a dataset has no group the
group links are simply absent.
- /groups/{g}/phewas with no parameters is a 422 ("give either locus, or
chr and mb"), so that link is now a shape with placeholders.
Placeholders stay where the caller must supply a value
(.../markers/{marker}/classes?trait={trait_id}), but the crawl now skips
them only when the description says what to fill in -- otherwise a client
reading the next list cannot tell a call from a template.
Two stale expectations alongside: /version carries the root links now (it
is the natural first call), and /search's results put its own follow-ups
first and the root links after them.
Tests are honest about their environment too: links skips when no search
index is configured (an empty XAPIAN_DB_PATH answers 502), and the
envelope crawl exempts /openapi.json, the API document, which is
deliberately not enveloped.
|
|
The GTEx problem -- it is human-only, and a mouse symbol there is an empty
report -- is answered by the endpoint that already knows which entity it
matched: /genes/{symbol}/aliases now returns
label the entity's canonical symbol in that species ("Shh" for mouse,
"SHH" for ?species=human) -- the symbol a human-only service needs
links [{name, href, description}] built for that symbol and species:
RGD, MGI, BioAlliance, NCBI Gene, Wikidata (the QID), UniProt,
STRING, GeneMANIA and GeneCup, plus GTEx/Open Targets/Protein
Atlas/GWAS Catalog for a human entity or BioGPS/Allen Brain Atlas
for a mouse one
So a mouse Shh reaches GTEx through the human ortholog's label rather than a
guessed one, and a client no longer holds the URL vocabulary. Only links
that answer are built: HomoloGene, UniGene, Genotation, GeneBridge and
PhenoGen are gone, and Panther is browser-only (see SKILL.md "Gene
link-outs", where each URL was checked).
A cached entity written before the label was kept is refreshed rather than
served -- otherwise the link-outs would be built on a symbol it does not
have -- and `stale: true` still says when Wikidata was unreachable and the
cached answer was used.
tests/wikidata.rs: a mouse gene gets the mouse services and no GTEx; the
human ortholog gets label SHH and a GTEx href built from it. Both halves
skip when the reply is stale, since a Wikidata timeout is not a failure.
|
|
Checked what the plot actually draws: it has never drawn the alpha 0.10
line -- the plotter matches 0.05 and 0.01 only. What lists all three is the
reply's notes ("alpha 0.01: LOD 4.58, alpha 0.05: 3.65, alpha 0.1: 3.50"),
which is where the 0.10 came from.
Still, the point stands on its own: 0.10 is the loosest bar in the set and
drawing it invites reading it as the line to clear. So the defaults are
stated and pinned -- 0.05 and 0.01 on the plot, 0.10 only on request
(?all_thresholds=true, or an explicit ?thresholds=) -- alongside GEMMA's
Bonferroni, which is opt-in for the same reason: a reference line is an
annotation, not a result.
tests/bulklmm.rs counts the lines: exactly two by default, and no
alpha 0.10 label.
|
|
A line on a plot is an annotation, not part of the result, so the default
is now no line at all. GEMMA's Bonferroni becomes opt-in like any other:
?bonferroni=true GEMMA's own line, labelled
?thresholds=3.65&threshold_labels=… another engine's, labelled
(nothing) just the scan
Which means a reader sees exactly the lines someone chose, each with its
source in the label -- and the accidental "this plot says significant"
impression of a default convention line is gone. The plotter's summary and
its <desc> no longer mention Bonferroni unless it was drawn.
tests/plot.rs: the plain plot has neither line; with thresholds the two
BulkLMM lines are labelled and Bonferroni is absent; with ?bonferroni=true
GEMMA's own line appears.
|
|
GEMMA has no permutation test here, so its plot carries a Bonferroni line.
A threshold from /compute/bulklmm belongs to a different null and must not
look like GEMMA's, so it is passed in and named:
POST /plot/gemma?thresholds=3.50,3.65
&threshold_labels=bulklmm%20alpha%200.10,bulklmm%20alpha%200.05
Each is drawn dashed in the series colour with "label: value" beside the
line, with GEMMA's Bonferroni line still there, and the plotter's
x-plot-summary names the reference lines it drew. A threshold that is not
a positive number is a 422 invalid_threshold rather than a silent omission.
/compute/bulklmm's reply offers that href already, with its alpha 0.05
threshold filled in -- so "same values, two models, one picture" is one
link from the scan, and the labels keep the two nulls apart.
tests/plot.rs: the plain plot has no bulklmm line; with thresholds both
are drawn and labelled; a non-numeric threshold is refused.
|
|
The marker blocks a single-trait scan is split into were the driver's
default (Threads.nthreads()) and changed with --threads. They are a
separate knob now:
BULKLMM_THREADS Julia threads (--threads) default 4
BULKLMM_BLOCKS independent marker blocks, scanned default 4
in parallel (1 = serial scan)
Blocks are exact either way -- LOD 9.589938 and the same permutation
thresholds at 1 and 4 -- so the knob only buys wall clock. The reply says
what ran ("split into 4 independent blocks and scanned on 4 Julia
thread(s) (BULKLMM_BLOCKS / BULKLMM_THREADS ...)"), because the answer to
"is it multicore?" should be in the response and not in the log.
|
|
We do not run GEMMA permutations -- each repeats the whole LOCO scan and the
gate gives GEMMA one slot -- but "not supported here" is not the same as
"not available". The refusal now says both:
1. POST /compute/bulklmm with nperm: 1000 -- an LMM whose thresholds come
from rotations that keep the kinship (R/qtl's nperm does not, and is
anti-conservative on a structured panel);
2. run GEMMA's own permutation locally. The reply's notes already name the
bimbam pair and the phenotype file, so `gemma-wrapper --permutate` or
`gemma -lmm 9 --permutate` on those inputs reproduces exactly the null
GEMMA would have used -- hours instead of seconds, which is the honest
price of that particular null.
docs/bulklmm.md gains the table this belongs in: three ways to get a
threshold, ordered by how honest the null is for a structured panel
(rotation with kinship; GEMMA's own LOCO permutation run locally; R/qtl's
nperm as a sanity check only). SKILL.md says the same in two lines.
tests/gemma.rs now asserts both halves of the refusal, and the earlier
assertion on /compute/rqtl -- which stopped being true when the refusal was
rewritten -- is replaced rather than left passing by accident.
|
|
docs/bulklmm.md records the three things that are easy to get wrong when a
BulkLMM result is quoted:
* where it stands against the other engines -- Haley-Knott with no
kinship (R/qtl), LOCO LMM whose permutations are refused (GEMMA),
LOCO LMM on HMM probabilities (R/qtl2), and this: a global-kinship
LMM whose permutation thresholds are rotation-based and therefore
cheap -- the only engine here that can say "genome-wide significant
at alpha 0.05" against a null that respects relatedness;
* what it is not: global kinship rather than LOCO, so its p-values are
not GEMMA's; h2_null says how much relatedness it found (near zero
here, which is why it reproduces R/qtl's 9.59 where GEMMA differs);
one degree of freedom, so multiparent scans stay with /compute/qtl2;
and no standard errors in 1.2.2's scan(), so the rows are
marker, chr, pos, lod, beta, h2;
* the cores, measured: Threads.@threads is in bulkscan.jl and
bulkscan_helpers.jl, never in scan.jl, so a single-trait scan is
serial by design and the visible parallelism was BLAS (96 threads
10.5 s against 4 threads 7.7 s). What the driver does about it --
exact marker blocks, cached matrix and kinship, capped BLAS, and the
breakdown printed into the reply -- with the numbers (20.9 s at one
block against 18.3 s at four, same LOD 9.589938).
Also the packaging picture, since it is not obvious from the outside:
guix-bioinformatics has a working julia-bin and a julia-build-system, but
only three of BulkLMM's closure are packaged, so a pure-Guix build is
35-45 recipes; the pragmatic route is julia-bin plus a depot built once
from the pinned Manifest. scripts/prepare-julia-depot.sh is that step,
written down, and the depot stays small and GPU-free (79 packages,
139 MB) with JULIA_PKG_OFFLINE=true at request time.
The plot test learns that both thresholds are drawn, not one.
|
|
Reported: a BulkLMM run does not look multicore. It is not a
misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint
calls -- has no `Threads.@threads` at all; the package threads
`bulkscan()` (many traits) and bulkscan_helpers, not scan. The
parallelism that was visible before was BLAS (OpenBLAS defaulting to all
96 cores here), which is why `user` time ran 4x `real` while Julia
threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s.
* The driver splits the markers into independent blocks and scans them
on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact:
the LOD for each marker is independent given the null variance
components, which depend on y and K only. Verified identical -- peak
LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks --
and permutation thresholds combine by taking each permutation's
maximum across blocks before the quantiles, which is what a
genome-wide threshold is.
* The parsed matrix and the kinship are cached per panel (read 1.3 s ->
0.4 s, kinship 0.9 s -> 0), because both depend only on the cross.
* The summary in `notes` now says what ran: julia threads, blocks, BLAS
threads, and the read/scan split.
* The hand parser is back: CSV.jl into a DataFrame is an order of
magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s).
* BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for
one scan).
Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks
18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of
the run and each block re-estimates its variance components. Where the
cores pay is bulkscan() -- many traits in one process, threaded over
trait blocks -- which is the shape the next endpoint wants; documented in
the README rather than left as folklore.
|
|
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values
have no threshold we can compute: a GEMMA permutation repeats the whole
scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual
stand-in and it is the wrong null -- it shuffles phenotypes as if the
individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global
kinship matrix and permutes by rotating the data through the kinship's
eigen decomposition, so a threshold costs seconds.
POST /compute/bulklmm {group, genotype_version, samples|basket,
nperm, seed, reml, assumption}
Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8,
35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2
4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81,
0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's
published locus and LOD, and a significant peak against a threshold this
API could not compute before.
Two differences from GEMMA are in every reply, not implied: a global
kinship matrix rather than leave-one-chromosome-out, and h2_null saying
how much relatedness was found (here ~0, so the fit is effectively
fixed-effects -- which is why it agrees with R/qtl where GEMMA differs).
* src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a
strains x markers matrix (our BIMBAM pair is already the allele
dosages BulkLMM wants), a marker map, the phenotype and covariates
-- runs the driver through streaming::run_streaming_with_env, and
reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid
f64 in Rust and would otherwise be reported as a value.
* scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds),
with the seed recorded so a threshold set is reproducible.
* gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is
cheap, unlike R/qtl's.
* Julia comes from the deployment's own store binary with a prepared
depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request
runs JULIA_PKG_OFFLINE=true, so a scan can never download anything.
`--project=<path>`, not `--project <path>`: Julia 1.8 takes the
space-separated form as a program argument.
tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and
seed are in the notes, and a scan without permutations says there is no
threshold rather than leaving a bare LOD to be read as significant.
|
|
Two engines reading one riset panel can fit different predictors, and the
reply now says which one it used and what it cost -- with this panel's own
numbers.
* `genotype_model: "riself" | "f2"` on /compute/qtl2 (`het` kept as the
older spelling: drop/keep). riself is the default: R/qtl's
convert2riself coding, the H calls become missing and the HMM
re-imputes them. f2 keeps them as a third genotype -- a diagnostic.
* The conversion now counts the calls it drops and keeps the count
beside the cross (conversion.json), so a reply can say
"610684 heterozygous call(s) (12.7 %) were dropped". A cross
converted before that file existed measures the count once, on first
use, rather than never.
* Crossed with the note that already explained the H drop, so the two
read together: what riself does, how many calls it cost, and what f2
would give.
Documented in the README as "Why R/qtl and R/qtl2 disagree on a riset
panel", with the measurements (BXD trait 16627, BXD.8: R/qtl 9.59 at
rs33299937; qtl2 riself 3.73; qtl2 f2 9.68), R/qtl's adaptive
convert2riself rule quoted from its own source, and qtl2's side: riself
has two states, f2 three, and `genril2` -- from qtl2 0.18, "the progeny
are inbred lines", cross_info first column the number of generations --
is the honest model for residual heterozygosity and is available in the
0.42 we run. What is missing for it is recorded precisely: a founder-based
cross expects individual genotypes coded as founder alleles rather than
1/2/3 (a prototype with numeric codes read every genotype as missing), so
founder_geno from the two parental alleles and an allele-coded het remain
to be done, with DOL as the counter-example (its H is a collapsed DO, not
residual heterozygosity).
tests/qtl2.rs: the model is reported (Cross: riself / Cross: f2), the
dropped count is in the reply, keeping the calls finds more, and an
unknown model is refused rather than ignored.
|
|
Returning genotypes is the one thing this API otherwise never does, so it
is fenced and the fence is the feature:
GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker>
* the server re-runs the R/qtl scan for the trait -- the same machinery
and gate as /compute/rqtl -- and takes the peak from that run, rather
than trusting a client that names a marker;
* the marker must lie within ?window= Mb of that peak (default 10), or
the answer is 422 not_the_peak quoting the peak's marker, chromosome,
Mb and LOD;
* the peak must reach ?min_lod= (default 3), or 422 not_a_hit;
* only animals that have a value for the trait are listed, and only
this marker's calls. Nothing else of the matrix leaves.
On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals
across A 110 / B 8 / H 74 -- the two a class of eight being exactly the
thing worth seeing.
POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's
`animals` (or its `genotypes` map) with the trait's values and get an SVG
with one point per animal, genotype class across, value up, class means
marked. Nothing is recomputed -- the plot is of the numbers the client
sends, like the other plots. New script scripts/plot-genotypes.py
(stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key.
tests/peakgenotypes.rs: the peak's calls for animals with values (with the
counts agreeing with the rows), the SVG, and a marker on another
chromosome refused with the peak quoted.
Also keeps /compute/rqtl's error order: naming both a group and a file is
"ambiguous" even when the request also forgot its values.
|
|
Two things asked for, and one bug each.
* POST /uploads/genotypes with an existing name now REPLACES it. A
genotype is the user's file to keep, not a record the deployment
curates; the digest in every reply says which version ran.
* genotype_upload on all four engines -- /compute/{rqtl,gemma,qtl2,
fitqtl} -- in place of genotype_file/group. The server writes the
form each engine reads: a GN .geno for R/qtl and R/qtl2, a BIMBAM
pair (new: genoupload::to_bimbam, coding from the base's
@mat/@het/@pat, NA for missing, snps as marker/bp/chr) for GEMMA. No
user path travels with a request.
Verified against the full 3811-marker BXD.4 matrix uploaded as an update:
rqtl 3811 rows, gemma 3809, qtl2 3730, fitqtl by marker.
The bug worth naming: values given by name were being aligned against a
file, and for GEMMA that file is BIMBAM, which has no individual names --
so the phenotype file came out empty and the wrapper failed with a Ruby
backtrace that said nothing. GEMMA's values are now ordered by the
upload's own individual list before the pair is written. The related
fixes: rqtl and fitqtl align named values onto the file's individuals
(they previously demanded its column order), and genotypes::align_to_file
is the one place that does it.
tests/genoupload.rs: mapping an upload on rqtl/qtl2/fitqtl with values by
name, and replacing a name (one row in the listing, a new digest). GEMMA's
LOCO kinship needs more markers per chromosome than a small fixture has,
so it is exercised against a full panel rather than in the fixture.
|
|
Tab-delimited, header names the individuals from column 4, then marker,
chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own
SQLite file -- nothing goes near MariaDB -- and written out on demand as
a GN .geno carrying the base's @-metadata, so the R scripts read exactly
what was uploaded.
An upload updates a population, it does not invent one, and the checks
say so:
* the base version must exist (resolved through the same
genotypes::resolve_named the rest of the API uses);
* at most 50 individuals the base does not have, and at least one that
it does -- no overlap is a different population, not an update;
* at most 200 000 markers, and the refusal says the sysadmin has to
organize a file that size, because this is a request path, not an
ingest;
* the values must be the same kind as the base's: letters, whole
numbers or probabilities. A letters panel holding 0/1 would be read
as the wrong thing silently, so it is refused naming both kinds.
GET /uploads/genotypes lists them, /{name} its metadata (overlap, new
individuals, kind, digest), /{name}/markers pages the marker table, and
/{name}/geno writes the file and returns the path for `genotype_file` on
the compute endpoints. DELETE removes one.
scripts/example-genotype-upload.py turns any .geno into a valid update --
a subset of markers, a few individuals dropped and added -- and
tests/genoupload.rs builds its own from the deployment's BXD.4 for the
same reason: upload, page the markers, write the .geno and check the
layout, delete, then break each rule (no overlap, numbers in a letters
panel, commas) and check the refusal says which.
Unit tests cover the three value kinds, missing cells, ragged rows, a
non-numeric position, the 50-new-individual limit, and the .geno layout
(Chr first, then the marker -- GN's order, not the upload's).
|
|
The other half of changing data: bringing new columns in. A matrix is a
delimited table -- a header row, then one row per individual, the first
column naming them -- stored in the uploader's own SQLite file and checked
against an existing genotype version's panel before it is accepted.
* POST /matrices {name, genotype_version, contents, kinds?}: parse the
table (tab, comma or semicolon, guessed from the header), then check
it against that version's individuals. The count must match and every
name must be in the panel; 92 of 93, or a name the panel does not
have, is a 422 that says which. A mapping that silently drops
individuals is worse than an error.
* kinds marks columns that are covariates; everything else is a trait.
* Each column becomes a basket named <matrix>.<column> -- a dot, not a
slash, because a basket name has to survive a URL path segment -- so
a column maps on any engine with {"basket": "my-traits.bw"} and
inherits the revision history and digest without a line of new
mapping code.
* GET /matrices lists them, GET /matrices/{name} their columns and
basket names, GET /matrices/{name}/values the table back (parsed
values, null where missing), DELETE removes the matrix and its
columns.
Two bugs this found in the basket store, both real:
* SQLite has foreign keys OFF by default, so the ON DELETE CASCADE
never fired: deleting a basket left its rows behind, rowids were
reused, and the next create hit "UNIQUE constraint failed:
basket_version.basket_id, revision". The pragma is now set, and
deletion removes the dependants explicitly, because files written
before it have orphans.
* The first attempt named a column's basket "<matrix>/<column>", which
the basket-name rule (rightly) refuses. The dot keeps both rules.
scripts/example-matrix.py writes a table from a genotype file's own header
with deterministic values, so the fixture passes the panel check by
construction -- tests/matrices.rs builds its own for the same reason and
walks the loop: upload, read back, map a column with R/qtl2, delete, plus
the three refusals (too few individuals, a name not in the panel, an
unknown version).
|
|
Three things the first cut was missing, and both were asked for.
* An automatic version history. Revision 1 is the column as first
stored and each PATCH adds one -- with its own digest and Unix
timestamp -- and earlier revisions are kept. GET
/baskets/{name}/revisions is the history, ?revision=N reads an old
column, and a mapping can pin one: {"basket": "wbc-edit@2"}. That is
the point of a digest: a result quoted yesterday can still be
reproduced after the column has moved on.
* Files named for the user, not a hash of the subject:
BASKET_DIR/<user>.sqlite (0600, WAL). Tokens rotate, subjects do not,
so a new token finds the same baskets -- and an operator can see
whose baskets a file holds. A subject that is not filename-safe is
sanitised and disambiguated with a short digest, and a pre-naming
hash file is renamed on first use rather than orphaned.
* /compute/rqtl and /compute/gemma and /compute/fitqtl take `basket`
too, so all four engines share one meaning: a basket resolves to the
same (name, value) pairs a client would have posted, and every reply
records the revision and digest it ran.
The schema upgrade is a real one, not a reset: an older file's single
column becomes revision 1 of the history, cells copied across with the
digest and timestamp they already had. Tested by building the old schema
by hand and opening it (an_older_file_is_upgraded_with_its_column_intact),
which is also how I found that the migration was recreating a table the
base schema had already made.
Verified on all four engines, head and pinned: rqtl 3811 rows, gemma
3809, qtl2 3730, fitqtl 3 markers -- each reply carrying "Basket '...'
revision N (digest ...)".
|
|
The next phase is changing data, and the constraint is that MariaDB is
never written to. A basket is the storage half of that: take a trait's
values, edit some of them (or build a covariate column that does not
exist), keep the column under a name, and map that instead.
* POST /baskets stores a complete column; GET /baskets lists yours;
GET /baskets/{name} its metadata; GET /baskets/{name}/values the
cells; PATCH /baskets/{name} replaces the column with a new revision;
DELETE removes it.
* POST /compute/qtl2 takes "basket": "<name>" in place of `samples`,
and both at once is a 422 rather than a silent choice. The mapping
code below it is untouched -- a basket resolves to the same
(name, value) pairs a client would have posted.
* One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600,
WAL, busy_timeout. `GET /baskets` only ever lists the caller's own;
the file name is a hash, so a directory listing does not name users.
* The full column is stored, not a diff -- the caller has the original
beside it, and a mapping needs one complete column.
* Text with a parsed copy beside it: raw (what was typed), value (the
number or NULL), status (ok/missing/bad). "NA" and 0 stay different
claims, and a column with something that is neither a number nor a
missing marker (NA, x, -, ., ?) is refused with the samples named --
silently dropping it would change a trait without saying so.
* A revision and a digest: an edit bumps the revision and changes the
digest, and a mapping reply records both ("Basket 'wbc-edit'
revision 2 (digest 9559a70e...)"), so two runs on one name cannot
look alike. A result cache must key on the digest, not the name.
The loop, end to end on BXD trait 10001: 34 values posted as a basket,
one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2
-- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the
revision and digest.
tests/baskets.rs walks that loop (create, list, values, patch, map by
name, delete) and the refusals: a non-number named in the error, a name
that cannot become a path, an unknown basket, and basket-plus-samples.
Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest
changing on edit, per-user isolation, and name validation.
rqtl/gemma take the same 10-line change next -- their `samples` shapes
differ, so they are a separate commit rather than a rushed one.
|
|
The gotcha notes in SKILL.md tell an agent to count the animals per
genotype class before believing a peak; a note in a document is only
found by whoever reads it, so the replies now say it too, at the peak
they just computed.
* suggest::class_hint builds
/groups/{g}/genotypes/{v}/markers/{peak}/classes?trait={trait_id} --
a real GET with one placeholder, because a scan knows the trait's
values, not its PublishXRef.Id.
* /compute/{rqtl,gemma,qtl2,fitqtl} add it to `next` after the PheWAS
link, naming the peak each reply found and the version the request
named (a {version} placeholder when it named none).
* The genotype subtree carries the template too: a version's detail,
and a concrete marker's own links.
* /compute/qtl2 gains the sentence that explains the divergence: on a
riself (riset) panel heterozygous calls are dropped before the scan
(R/qtl's convert2riself) and the HMM re-imputes them, while GEMMA
keeps them at dosage 0.5, so a mostly-H marker cannot peak there and
two engines can name different markers of one region without either
being wrong.
tests/links.rs: a scan's reply offers the class counts at its own peak --
the link names the version and the peak marker the reply reports, and
leaves {trait_id} for the caller.
|
|
The question that follows "why do GEMMA and R/qtl2 peak at different
markers": what does the trait look like within each genotype class at one
marker. The frequencies endpoints report the distribution of calls, not
the values behind them, and the answer must not be "serve the matrix" --
this API never does.
GET /groups/{group}/genotypes/{version}/markers/{marker}/classes?trait=<id>
-> {"A": {"n", "mean", "sd", "min", "max"}, "H": {...}, "B": {...}}
The calls are joined to the trait's per-strain values inside the server
(the same lookup, authorization and 404s as /groups/{g}/traits/{id}/values)
and reduced to those five numbers per class. Nothing in the payload names
an animal. A class with fewer than ten animals is called out in notes,
because a peak resting on two is a different claim from one resting on a
hundred.
It settles the DOL trait 10003 question. At GEMMA's top marker
JAX00418132 (chr16 36.35 Mb) the classes are A n=2 mean 1.8, H n=20 mean
2.53, B n=171 mean 3.95 -- the A class is two animals and they are the
trait's two lowest values. R/qtl2 drops H on a riself panel (R/qtl's
convert2riself), so it keeps 173 animals there but only 2 of one class:
nearly monomorphic, which is why its scan peaks elsewhere. GEMMA keeps
the 20 hets at dosage 0.5, and that is where its sharper p-value comes
from. GEMMA's next two hits are the same pattern in opposite directions
(one A among 171, then two B among 171), so the chr16 block is a haplotype
carried by a handful of extreme animals, not a population effect.
Also fixes the reading of GEMMA's `af`: it is mean(dosage)/2 under the
BIMBAM coding this deployment ships (B=1, H=0.5, A=0), so 0.469 for
JAX00418132 is half the B-allele frequency 0.935, not the A-allele
frequency 0.065. The bimbam file and DOL.geno agree at that marker.
tests/frequencies.rs gains a route test: the class fields, A < B, the
small-class note, the "not served" note, no individual names anywhere in
the payload, and a 404 for a trait the group does not have.
|
|
A GeneNetwork trait page shows the gene's aliases next to the symbol,
and GN4 had no way to ask for them: gn-guile has a Wikidata lookup
(gn/db/sources/wikidata.scm) and nothing in GN4 did. This is that
lookup with the caching Wikidata deserves -- it is a shared public
service and an alias list is not news.
* GET /genes/{symbol}/aliases: two SPARQL queries against
WIKIDATA_ENDPOINT (the entity: a gene wdt:P31 wd:Q7187 in human,
mouse or rat, labelled with the symbol; then its skos:altLabel s in
English), cached in WIKIDATA_CACHE_DIR/gene-aliases.lmdb, an LMDB
file (NO_SUB_DIR, 0600, so one file and LMDB's -lock companion
rather than a directory of two).
* TTL WIKIDATA_TTL_SECONDS, seven days by default. Past it the value
is refetched; ?refresh=true forces that.
* The label is matched case-insensitively, which the gn-guile query
did not: human genes are labelled SHH and mouse ones Shh, so an
exact match finds only one species. ?species=human now returns the
ortholog's labels -- HLP3, HPE3 and the rest -- which are the ones a
mouse trait page shows. With no species named the pick is
deterministic (mouse, rat, human), so a repeated request cannot
change entity.
* A failed refresh keeps the last value: the answer says stale: true,
the envelope is partial, and a note explains it. Nothing cached at
all is a 502 that says so. An empty alias list and an unreachable
upstream are different facts, and only one of them is about the
gene.
Seven unit tests (key scoping, TTL including a clock that went backwards,
the query shape and quote-escaping, SPARQL flattening, an LMDB
round-trip that asserts the file is gene-aliases.lmdb, and
stale-on-failure with and without a cache) and two route tests (the
second call is served from the cache -- same fetched_at -- and the
species switch), which skip with a 502 assertion when there is no
network.
|
|
A probeset id is not a Xapian field, so /search cannot find 1427571_at
exactly and the only route to it was guessing a dataset and calling
/datasets/{ds}/probesets/{name} until one answered. A gene symbol could
be searched, but the reply was capped at 100 hits, interleaved probesets
and platforms, and carried no total -- so "did I see all of Shh?" had no
answer. Both questions are one indexed query each: ProbeSet.Name_IDX and
symbol_IDX.
* GET /probesets/{name}: one row per dataset that carries the probeset,
with the panel (group, species, platform) and the scan's statistics
(mean, LRS, LOD, locus, additive). Suffixed chip variants match too
-- 1427571_at also finds 1427571_at_A -- and each row's `name` says
which matched, so the answer never hides a substitution. LIKE's
wildcards are escaped, or every Affymetrix `_at` would match any
character. 155 rows over 155 datasets in 32 ms.
* GET /genes/{symbol}/traits: the same rows filtered on Symbol, with a
real total, ?sort=lod, ?min_lod=, ?group=, ?species=, ?platform= and
paging -- "BXD datasets where Shh is a strong eQTL" is one call.
1225 rows over 17 datasets and 13 probesets in 38 ms.
* Both serve metadata, not values. A token is enough for the index;
trait values stay per-dataset and authorized per dataset, and `next`
goes to /datasets/{ds}/probesets/{name}/values, which checks. There
is no cross-dataset values endpoint, deliberately: that is bulk work
and /groups/{g}/traits is the bulk call.
The one-statement join took 5.2 s for Shh -- MySQL rescans
InbredSet/Species per row -- so the 933 ProbeSetFreeze rows and the 102
panels are fetched separately and joined in memory: 70 ms each. nsamples
is left out: it is a ProbeSetData count per row (+0.5 s) and the dataset
record already reports strains.
|
|
A caller that knows only /version should be able to reach anything by
following links. Until now `next` was thin and uneven: the genotype
subtree offered none at all, the group record offered one, /version
offered nothing, and two links it did offer could not be followed.
* suggest:: now builds the graph: root_links (groups, species,
mapping-methods, openapi), genotype_version_links (frequencies,
marker names, a marker's location and tally, the group, the scanning
actions), unlisted_genotype_links, marker_links (location, tally,
marker list, panel tally, PheWAS at that marker), group_links and
trait_links. Envelope::with_links / with_links_except add them,
deduplicating and never suggesting the response the caller is
already reading.
* Wired through the genotype surface (listing, version detail, markers,
location), both frequencies routes, the compute endpoints, groups,
traits, both value routes, species, datasets, publications,
accessions, search, mapping methods, phewas, correlation and
/version.
* Fixed what was not followable, because `next` promises a GET that
returns 200 and a client is the one that finds out otherwise:
- /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are
gone; comput endpoints now link ?locus=<their strongest marker>
(suggest::peak_marker_of, computed before the rows move).
- /datasets and /publications are not routes; root_links offers the
listings that exist (/groups, /species, /mapping-methods) and
/openapi.json.
- /search?q= with an empty query is a 400, so it is not a link.
- the correlation endpoint's "/compute/correlation" was a POST in
`next`; it is in `actions` now.
- marker_links pointed at /markers/{marker}, which is not a route;
it points at /markers/{marker}/location.
tests/links.rs walks the graph two hops from a dozen seeds and fails on
any link that does not answer 200, plus a check that a tally leads to the
marker list and to a marker's location, and a location to that marker's
tally and its PheWAS.
|
|
Two questions the genotype surface could not answer: what a panel is
actually called as, and which markers it holds. The listing gives counts
and the compute endpoints give scans; a panel whose calls are not the
codes its header declares, or a marker that is 99% missing, stayed
invisible until a scan came back flat -- and then the search for the
cause started at the wrong end.
* GET /groups/{group}/genotypes/{version}/frequencies counts the calls
in the version's .geno and returns them as call -> frequency
({"B": 0.4889, "D": 0.492109, "H": 0.018991}). Frequencies are over
the calls that are present, so they sum to 1; missing codes (U, N,
NA, an empty cell) are counted separately and reported in notes,
because the missing rate is the diagnosis.
* A probability-valued file is binned 0.05 wide (twenty bins over
[0, 1], keys are the bin edges), since 0.4187 and 0.4191 are the same
statement about a marker; ?bin= sets another width. A file with a few
distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported
exactly -- binning three values hides what is being looked at.
* .../markers/{marker}/frequencies is one locus, and ?marker= on the
panel route is the same call. Counting goes through an interner, so
cost is one hash lookup per cell and one allocation per distinct
call; a .geno over 256 MB is refused (413 genotype_too_large) rather
than read, which is what HSNIH-Palmer_true (3.7 GB, 134927 markers x
6147 individuals) demanded.
* The version's /markers now lists marker NAMES only, one column, and
.../markers/{marker}/location returns {"name","chr","cM","Mb"} as
named fields: a page of names stays small, and a position is a
lookup rather than a table the client must hold. The same size guard
applies, so an oversized panel's listing is a 413 in 3 ms.
* Files no group lists -- a panel in preparation, a fixture, the DO's
do/ cross -- live at /groups/unlisted/genotypes/{name}/... with the
same four calls. `unlisted` is a namespace, not a group: only its
genotype subtree answers. One prefix for the whole genotype surface
beats a second spelling for the same tally, so the earlier
/genotypes/{name} spelling is gone.
* genotypes::resolve_named (moved out of the frequencies module, since
it is genotype metadata) resolves a version, a .geno name, or a path
relative to the genotype files directory; MAX_SCAN_BYTES is shared by
every whole-file read.
Tests limited to the feature: genotypes 3 (names-only columns, paging,
location fields, marker_not_found, unlisted parity, 413), frequencies 5,
frequencies unit 7.
|
|
A multiparent population cannot come from a .geno. That file carries two
codes per marker (B/D, A/H/B) -- a collapse, not a founder allele -- and
the collapse is not invertible, so DOL's 194 DO animals cannot be fitted
as a DO however much reference data appears. What can be is a cross that
ships its own qtl2 control file with founder genotypes, so the endpoint
now takes one by name:
{"rqtl2_cross": "do", "trait_name": "DO::10001 (WBC)", "samples": ...}
* resolve_control_file looks in genotype/rqtl2/{name}.json, then
{name}/{name}.json (do/do.json), then {name}.json under
GENOTYPE_FILES; prepare_control_cross reads the cross's pheno, maps
and founder_geno headers without converting anything. It reports
crosstype, founders and whether the cross is multiparent, and
refuses a multiparent crosstype with no founder_geno rather than
silently scanning it as biallelic.
* Samples are matched to the cross's own population by name and the
strays are counted in notes; rqtl2_cross cannot be combined with
group or genotype_file.
* The gate is given individuals x founders, because a multiparent scan
fits one coefficient per founder allele at every marker.
* Autosomes only now applies to shipped crosses as well: the DO's X is
left out (369 markers) instead of being fitted without a sex
covariate, and the script subsets with cross[, chr = autosomal].
Measured on the DO (WBC, 742 individuals, 8 founders, 6044 autosomal
markers): probabilities 174 s / 1.39 GB (cached), LOCO kinship 15 s
(cached), scan 21 s. scan1 runs on one core for a multiparent cross
because its forked path stalls on allele probabilities here -- 25+
minutes at 8 cores with no CPU consumed, against 21 s serial.
Two bugs found by running an actual published trait through it:
* do_pmap.csv is in Mb, not the bp qtl2 documents, so every DO peak was
reported at 0.0 Mb. The map's unit is now detected from its largest
position and printed.
* The phenotype file was read with the default whitespace separator, so
a label with a space ("DO::10001 (WBC)") split into three fields and
read.table stopped with "line 1 did not have 3 elements". Both the
pheno and covariate reads are tab-separated now.
The R error extraction also joins R's wrapped continuation line, so a
complaint reads as a sentence rather than "Error in scan(...) :".
|
|
R/qtl sits between the existing two scans: Haley-Knott on marker
genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with
mean-imputed genotypes. Neither can analyse a multiparent population,
and neither models relatedness from the actual genotype data. R/qtl2
does both -- HMM genotype probabilities, then an LMM with
leave-one-chromosome-out kinship -- and it is what names the third peak
on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24,
against GEMMA's -log10 p 7.15 there).
* src/qtl2.rs converts this deployment's own .geno into a qtl2 cross
under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with
the genotype-code map, the orientation and the paths written out
explicitly. The shipped genotype/rqtl2/ control files cannot be
used -- they point outside the data directory, declare no code map
(every genotype reads as missing) and no orientation (individuals
read as markers) -- and each of those produces a plausible flat
scan instead of an error, which is why the conversion is ours.
* Heterozygous calls are dropped on a riself panel, as R/qtl's
convert2riself does. Left in, qtl2 reads the cross as multiparent
and its scan1 takes a path the panel cannot satisfy ("Error in
scan1_pg ... : No individuals in common").
* Autosomes only: X needs a sex covariate to be fitted properly and
Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is
restricted the same way. The note says how many markers were left
out.
* scripts/qtl2_scan.R caches genotype probabilities and kinship by a
hash of the control file *and its directory*; two conversions of
the same panel can have identical control files and identical file
sizes, and reusing the first one's probabilities silently scans the
wrong cross. It also takes the individual names from whichever
shape the cross's pheno slot has -- a single-column pheno file
leaves it a plain matrix, and the resulting empty phenotype reads
as scan1's "No individuals in common".
* Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply
unchanged -- post the same samples as /compute/rqtl.
* tests/qtl2.rs: the fixture scans and its notes name the cross; BXD
trait 10001 peaks on chr8 under both scans; refusals carry
missing_genotype / invalid_kinship.
cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like
R/qtl; a DO cross costs minutes of genotype probabilities, paid once
per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
|
|
DOL (200 individuals, 112 913 markers, one genotype version) failed on
R/qtl with "There are missing chromosome IDs" and on GEMMA with a bare
wrapper bail-out. Two unrelated causes, both in our own code.
**R/qtl: a genotype code collided with a chromosome name.** DOL's header
says `@unk:X` -- X is the code for a missing genotype -- and the wrapper
passed that to `read.csv(na.strings=...)` for the *whole* file, so every
marker on the X chromosome lost its chromosome ID and `read.cross.csv`
refused the map. The unknown code now applies to the genotype columns
only. DOL scans: 112 913 markers in 42 s, where before it never got past
reading the file. Same fix in scripts/qtl_fitqtl.R, which shares the read.
**GEMMA: its leave-one-chromosome-out run cannot fit the Y.** With LOCO the
DOL panel is scanned per chromosome, and the Y run dies inside BLAS --
`Enforce failed for N>0 in src/fastblas.cpp ... fast_cblas_dgemm` -- taking
the whole request with it (a 502 whose detail is the wrapper's banner).
The mitochondrial chromosome has the same shape of problem. `prepare` for
the GEMMA engine now writes the bimbam pair without the Y and M markers
(cached beside the individual subset, line-aligned so the geno and snps
stay in step), and the note says what was left out: "46 Y/M markers left
out (GEMMA's LOCO cannot fit them)". DOL maps: 106 515 rows in 49 s.
**And the four tests that had been failing since before this work**, all
stale expectations rather than product bugs:
* `benchmark_questions` pinned two traits to "page 2" of a 100-per-page
search when they rank ~190th and ~219th of 246 -- one re-index pushes
them over. It now walks up to three pages and asserts presence and LOD,
not the page; and the helper `search()` pages, because a benchmark row
can name a trait that ranks outside the first hundred of its own query.
* `no_db_mode_degrades_gracefully` asserted exactly one hit for "alcohol"
from the index alone. The index grew; the behaviour under test is that
the search still answers, within the limit, with the database-backed
fields null. That is what it asserts now.
* `search_proxy_scenarios` and `search_parity_replay` never pinned the
engine they test, so an ambient `SEARCH_ENGINE=native` made the proxy
scenarios answer from the local index: a 200 where a 422 was expected,
and "expected 2 upstream calls, got 0" for every replayed query. Both now
set `SEARCH_ENGINE=forward`.
* src/genotypes.rs: `DEGENERATE_CHROMOSOMES`, `drop_degenerate_chromosomes`,
`Prepared.markers_dropped`.
* src/routes/gemma.rs: also the 500 that `eb5320d` introduced -- counting
parents/F1s from `panel - keep` instead of subtracting the exclusion
*set*, whose length can exceed the panel (subtract overflow).
* scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: na.strings scoped to the
genotype columns.
* tests/{benchmark,no_db,search,search_parity}.rs: as above.
Verified through the API on DOL trait 10003: R/qtl 112 913 rows, GEMMA
106 515 rows. gemma/rqtl/fitqtl suites green (14, 10, 4).
|
|
Asking whether the singular matrices could be fixed turned out to have two
answers, one of them a real bug on our side.
The bug: a panel can name individuals whose genotype row is empty, and an
empty row makes the kinship matrix singular. GEMMA then reports
"eigenvalues close to zero" and dies in LAPACK with
`LUDecomp ... input domain error` -- a message about linear algebra for a
problem that is a missing genotype. Measured on this deployment's own files:
BXD-AE.4 136 of 344 individuals have no genotypes
MDP 93 of 178
BXD-Heart-Metals.4 200 of 481
HET3-ITP.3 2867 of 2867 (the whole file is empty)
Dropping them makes those panels map: BXD-AE.4's GWA step goes from a
failure to 3.2 s, and MDP's kinship step from a failure to success. The
individuals contribute nothing to a scan either way, so nothing is lost;
`notes` counts them, and the "Genotype:" line no longer blames parents/F1s
for them (`prepared.excluded` is "panel minus kept", which now includes
both, so the two counts are separated).
Not the cause, but worth naming: `AXB` fails earlier still, on its
annotation -- `ERROR: Enforce failed for D1Mit296 in src/gemma_io.cpp ...
ReadFile_anno` -- which is a broken `.snps` file, not a matrix. That wants a
pre-flight check of the annotation (report the offending marker) rather
than a GEMMA failure, and is not in this commit.
Considered and rejected: a ridge on K (`K + epsilon*I`) would work
mathematically but GEMMA has no option for it and the wrapper caches K by
content hash, so it would mean patching the wrapper's cached matrix; `-gk 2`
and dropping markers do not add rank; and the non-LOCO path is the one that
does not run at all here.
* src/genotypes.rs: `individuals_with_genotypes`, one pass over the bimbam
file marking the individuals with any genotype at all.
* src/routes/gemma.rs: filter `keep` by it before alignment (so the trait
values drop with them), and say so in `notes`.
* tests/gemma.rs: `individuals_with_no_genotypes_are_left_out` -- BXD-AE.4
maps, the note names the drop and the reason, and the genotype line no
longer claims 344 individuals.
* README.md: the behaviour and why it exists, under /compute/gemma.
307 tests pass; the same four as before do not.
|
|
44 of this deployment's `.geno` files carry `cM` and no `Mb` column, and
none of them could be scanned. GeneNetwork3's wrapper (copied verbatim
here) assumes the position columns are `Chr, Locus, cM, Mb` and then starts
the samples at a hardcoded column 5. On a cM-only file that is one column
too far, so the genotype block loses its first individual and the whole scan
dies in `do.call(rbind, ...)` with "number of columns of result is not a
multiple of vector length" -- a message that says nothing about the cause.
The wrapper now picks the map column from whichever of the requested scale
and the file's own columns exists (falling back, and saying so), and starts
the samples right after the position columns; the sex default and the
sample-count guard use the same count. `scripts/qtl_fitqtl.R` had the same
assumption and got the same fix.
This is the second deliberate deviation from GeneNetwork3's script, after
the sample-count guard, and both are commented as such at the code.
* scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: the position column and the
first sample column are computed, not assumed.
* tests/test_data/rqtl_sample1_cm.geno: the existing fixture with `Mb`
dropped -- same panel, cM only.
* tests/rqtl.rs: `a_cm_only_genotype_file_maps` scans it and asserts all six
markers come back (the failure mode was five-vs-six columns, not an
error message that pointed at the genotype file).
* src/rqtl.rs, src/routes/rqtl.rs: the docs said the script never forwards
`--scale`; it does, and now falls back, so the request field says what
`pos` is in.
Verified through the API: DGRP (cM-only) now maps (3 markers), and AXB (499
markers) and BXD.4 (3 811) are unchanged.
307 tests pass; the same four as before do not.
|
|
`fitqtl` takes a `covar` argument, so the effect it reports can be the one
*after* conditioning -- which is usually the number worth quoting for a peak
that overlaps a known confounder. `covariates` now works on /compute/fitqtl
exactly as on /compute/rqtl: the same `covariates::resolve` (case attribute
by name, inline values, explicit map, same coding and same note), so the two
endpoints report a covariate identically.
The R side does what R/qtl needs: the phenotype file carries one column per
covariate after the trait, `--covarstruct` says which are categorical, and
the script dummy-expands those (k-1 against the first sorted level, missing
values left missing). Two things it has to get right:
* `model.matrix()` drops rows with a missing value, which silently misaligns
the covariate frame against the cross -- so the dummies are built directly
(`ifelse(is.na(x), NA, x == level)`), not through `model.matrix`;
* a covariate can be perfectly spread across the panel and **constant among
the individuals a trait covers** -- BXD's `Status` is exactly that -- and
R/qtl then stops with `Lapack routine dgesv: system is exactly singular`.
Columns that do not vary among the phenotyped individuals are left out,
and the reply names them, the same rule the GEMMA covariate file applies.
Measured on BXD trait 10001 at its peak, `BXD.4`:
no covariate LOD 2.9310 additive +2.394 32.8% var
with Status LOD 3.3602 additive +2.463 36.6% var
with an inline numeric covariate LOD 2.9373 additive +2.361
The LOD is no longer the scan's once a covariate is fitted -- it is a
different model -- which the docs now say, next to the no-covariate parity
check that still holds.
* scripts/qtl_fitqtl.R: `--covarstruct`, dummy expansion, the degenerate
column rule, and the dropped names in the summary line.
* src/fitqtl.rs: writes the covariates into the phenotype file and the
struct file, and passes both.
* src/routes/fitqtl.rs: the `covariates` field, resolved after alignment so
the coding and the complete-case count match /compute/rqtl.
* tests/fitqtl.rs: a covariate run (the note, the fitted row, a changed LOD)
and an unknown attribute refused with the list of valid ones.
* README.md, docs/www/SKILL.md: covariates on this endpoint, and why its LOD
stops matching the scan's.
306 tests pass; the same four as before do not.
|
|
`/compute/rqtl` answers "where" -- scanone gives a LOD per marker -- but not
"how much": a scan statistic is not a fitted model. R/qtl's `fitqtl` fits
one, and reports the QTL's effect, its standard error, the model LOD and the
variance explained. That is what a table of effects needs, and it is a
separate endpoint because it is a separate question, asked about a handful
of markers (a scan's peaks) rather than a genome: `fitqtl` fits one model
per marker, so a genome-wide version would be thousands of fits. Cap: 100
markers per request.
Rows are positional:
marker, chr, pos, lod, pct_var, additive, se, t, p
`additive` is fitqtl's estimate for the QTL term -- R/qtl codes the QTL
genotype +-1, so it is half the difference between the two homozygote
classes, the same convention GeneNetwork's published `additive` uses.
**Two independent checks that the numbers are right.** On BXD trait 10001
(`BXD.4`), at the scan's peak:
* `lod` = 2.930985, and `/compute/rqtl`'s scanone gives 2.930985 at the same
marker -- identical to every digit the CSV carries, because they are the
same cross and the same phenotype. That parity is now an integration test;
* `additive` = 2.394472, against the value genenetwork.org publishes for that
trait (`/api/v_pre1/trait/BXD/10001` -> 2.394444), which is 28 ppm away --
and that published number came from qtlreaper, a different engine.
* scripts/qtl_fitqtl.R (new): reads the same `.geno` and two-column
phenotype file as `rqtl_wrapper.R` (which is a verbatim copy of GN3's
script and so not extended), builds the cross the same way, fits
`fitqtl` at each named marker with `get.ests=TRUE`, and writes one CSV
row per marker -- plus a one-line summary naming the strongest fit.
`calc.genoprob(step=0)`: marker-level probabilities are all a
marker-level fit needs.
* src/fitqtl.rs (new): runs it, parses the CSV into positional rows (`chr`
stays text, it can be `X`), keeps the command line for `notes`.
* src/routes/fitqtl.rs (new): the endpoint -- same alignment rules as
`/compute/rqtl` (by name onto the panel, parents/F1s excluded unless
asked), the same `normalize`/`winsor_*` pipeline (effects come out on
that scale), and the same R/qtl gate, so a fitqtl request and a scan
queue against one another rather than competing for cores.
* src/config.rs: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`).
* tests/fitqtl.rs: the fixture (columns, sign of the effect, notes), the
LOD parity against scanone, and the refusals (no markers, both genotype
inputs).
* README.md, docs/www/SKILL.md: the endpoint, what each column means, and
when to reach for it instead of the scan.
305 tests pass; the same four as before do not.
|
|
Fitting `SeqCvge` as a covariate came back as `502 gemma_failed` with a
message about LU decomposition: its values are depths (`40`, `41`) with a
few two-library entries (`36+39L`), so the attribute was coded as **twenty
categories** and GEMMA died on the resulting dummies. The failure named
GEMMA, which is the wrong place to look.
`code_case_attribute` now refuses that shape at the covariate step, naming
the values and what to send instead:
422 invalid_covariate: case attribute 'SeqCvge': the attribute mixes
numbers with number-like text (14 of 19 values look numeric, but e.g.
'36+30L' does not), so coding it as categories would fit a quantity as a
factor -- and GEMMA fails on the result. Pass an explicit `map` giving
each of those a number (there are 5 such value(s): ...), or send the
numbers inline.
The rule is deliberately narrow: an odd level with *no* digits (`1, 2,
unknown`) is still a level and still coded, so nothing that worked stops
working; only dirty numbers are caught.
Also, the skill's covariate coverage -- which the review above found thin:
* when a covariate is worth trying (metadata that could confound: sex, age,
epoch, batch, sequencing depth), and the covariate step in the session
outline;
* read `.data.descriptions` before choosing one -- they are the only place
that says `SeqCvge` is sequencing depth and `Epoch` a family subgroup --
and the endpoint table now says so;
* read the covariate line in `notes` for the coding and the complete-case
n, and report that n beside any LOD/p change; plus `Cyro`/`Cryo` being
two levels in BXD;
* the categorical-vs-numeric rule measured rather than asserted: `RRID`
refused (82 distinct), `Status`/`Epoch` accepted, and the compound-value
trap written up with the `map` workaround;
* GEMMA's dummy columns, the column it drops when a level has no fitted
individual, and that R/qtl's note has no such line; missing values
dropping individuals; the 8-covariate cap and the `group` requirement.
* src/covariates.rs: the refusal, with a unit test for both sides of it;
* tests/gemma.rs: `SeqCvge` is a 422 naming `map`, not a GEMMA failure;
* docs/www/SKILL.md: the covariate section, the session line, and the
case-attributes rows in the endpoint table.
300 tests pass; the same four as before do not.
|
|
GeneNetwork2's trait page offers a set of normalisations, and its mapping
page warns about outliers without ever fixing them. This adds both, under
one spelling that `/compute/rqtl`, `/compute/gemma` and
`/groups/{g}/traits/{id}/values` all share:
normalize none | log2 | log10 | sqrt | negate | zscore |
robust_zscore | rank (GN2's names `invert` and
`qnorm` accepted)
winsor_quantile clip each tail at a fraction, e.g. 0.05
winsor_fences clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule
They run in a fixed, reported order -- transform, then winsorise, then
standardise -- so clipping acts on the scale being analysed and the result
lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The
note gives the steps *in the order they ran* (`winsorized at Tukey fences,
then zscore`), the count clipped in each tail, and how many values sit
outside Tukey fences -- the warning GN2 gives before mapping and this API
gave nowhere.
**Checked against genenetwork.org itself.** GN2's public trait page embeds
its own reference values (`js_data.qnorm_values` / `zscore_values`, the
`data-qnorm`/`data-zscore` attributes), so there is an oracle for the two
transforms whose definition matters. On BXD trait 10010 (28 samples):
`rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's
three-decimal rounding. Getting there found a real disagreement: GN2 uses
`scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use
the **population** sd; ours used the sample sd and came out
`sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2.
Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based
inverse-normal (Blom) transform, so it is called `rank` here; and its
`sqrt` silently adds 1 when a value is in [0,1], which shifts every effect
size -- a negative value is refused instead.
* src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom
scores over an Acklam inverse-normal, `Params` for query/body use,
`to_json` for suggestions, and eleven unit tests -- the quantile table,
ties sharing an average rank, missing values never imputed, GN2's own
`find_outliers` example, and the note's wording.
* src/routes/{rqtl,gemma}.rs: the three fields, applied **after**
alignment (over the samples that enter the fit) and before anything reads
the values, with the note; the R/qtl permutation suggestion and GEMMA's
R/qtl suggestion carry the pipeline.
* src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values
endpoint, and the mapping bodies in `actions` carry the same fields -- a
caller who fetches z-scores must map those numbers, not the raw ones.
* tests/groups.rs: values are mean 0 / population sd 1, the note says so,
every `/compute/*` suggestion in `actions` carries it, and an impossible
pipeline is a 422 `invalid_normalization`.
* README.md, docs/www/SKILL.md: what each name means, the order, the
population-sd parity, and that effects come out on the transformed scale.
298 tests pass; the same four as before do not.
|
|
A GEMMA row now carries the effect beside the p-value it belongs to:
`… beta, se, l_mle, additive, p_lrt`. `additive` is `-beta/2`, which is
what GeneNetwork's own tables and CSV call the additive effect
(gn2/wqflask/marker_regression/gemma_mapping.py:240). Halved because half
the difference between the two homozygotes is the usual additive effect;
negated because GN2's BIMBAM conversion codes `@mat` as 1, so GEMMA's beta
is per *maternal* copy while GeneNetwork's published sign is anchored on the
other allele. Two facts that make this worth doing carefully:
* `beta` and `se` are left exactly where GEMMA put them. `se` is what makes
the effect interpretable (`beta +/- 1.96*se` is a 95% interval) and both
are GEMMA's own numbers, so replacing them with a derived one would lose
information;
* the derived column goes *before* the p-value, not after it. `p_lrt` must
stay the last column: `gemma::peak_marker` reads `row.last()` for the peak
(and with it the PheWAS and plot suggestions), and the plotters and the
docs all say "the last column is the p-value". Appending would have
silently pointed every peak at an effect size.
The response also says what the three columns mean, and that they describe
*this run* on these samples and this genotype version -- not the `additive`
trait info reports, which came from the published mapping (trait 20010:
this run -0.54, published +2.118, same peak marker).
Nothing is added when GEMMA wrote no `beta` (a non-LMM run): the columns
are then exactly GEMMA's.
* src/gemma.rs: `with_effect_size`, applied by `parse_assoc_file`, so
/compute/gemma and /compute/gemma/ravanan both get it.
* src/routes/gemma.rs: the note.
* tests: four unit tests (placement, sign and halving, `peak_marker` still
finds the peak from the last column, a table without beta untouched, a
null effect for a row with `NA` beta) and one end-to-end test asserting
the column sits immediately before `p_lrt`, every value is `-beta/2`,
`p_lrt` stays a probability, and the note is there.
* README.md (the response example and what the column means),
docs/www/SKILL.md (how to read and quote it).
286 tests pass; the same four as before do not.
|
|
Two GEMMA runs over one BXD panel drew genuinely different pictures --
traces differ on all 20 chromosomes, peaks -log10 p 5.48 at chr8 97.0 Mb
against 5.43 at chr1 175.6 Mb -- and both came out headed
**"trait gemma: GEMMA association along the genome"**, because GEMMA's
request had no trait label at all: `/compute/gemma` fell back to
`<group>::gemma`, and `/plot/gemma` fell back to "gemma". The plot of each
trait was told apart by its annotation and nothing else.
`trait_name` is now an optional field on `GemmaRequest`, used for the
`Trait: ...` note (first in `notes`, as on `/compute/rqtl`), for the PheWAS
and R/qtl suggestions, and for the `/plot/gemma?trait=` suggestion -- so a
posted response titles its own plot, and `?trait=` still overrides.
Without it nothing changes: the label stays `<group>::gemma`.
* src/routes/gemma.rs: the field, the label resolution, the note, the
plot href.
* tests/gemma.rs: the covariate run names its trait and asserts the note is
first; tests/plot.rs: a GEMMA plot titles itself from the note, and an
explicit `?trait=` wins over it.
* README.md (`/compute/gemma`, and the plot section), docs/www/SKILL.md
(the GEMMA options list, and what the plot title comes from).
281 tests pass; the same four as before do not.
|
|
Answering "is robw using this successfully, and the others?" from the
access logs took an hour of inference -- user agents, search terms, which
traits were mapped -- because nginx records the address and the subject is
only in the token. Two callers can share an address (an agent and a person
behind one NAT), one caller can move between addresses (an LLM's egress is
a /24), and the answer for a given person was "probably 80.115.1.132".
The API now sets `x-gn4-user: <token sub>` on every **authenticated**
response, and the `gn4` log format records it as `user=`. The subject is
only known inside the handler (the `AuthToken` extractor verifies the
token there), so a middleware opens a slot in the request extensions, the
extractor fills it, and the header is set on the way out; a request with no
valid token gets no header, which is the honest answer and reads as
`user=-`. The caller only ever sees an identity it already presented.
* src/auth.rs: `USER_HEADER`, `CallerSlot`, and the extractor recording
its subject.
* src/lib.rs: the `tag_caller` middleware, outermost so the slot exists
before any extractor runs.
* tests/auth.rs: `an_authenticated_response_names_its_caller` -- the header
equals the token's `sub`, and `/version` (no identity) carries none.
* etc/nginx-vesuvio.pu.ac.ke.conf (and the deployed copy of it): `user=`
in the log format.
* README.md, docs/nginx.md (the format, an `awk` one-liner for per-user
counts, and the caveat that old and new lines differ in field count),
docs/www/operator-checklist.md (`user=-` means no token, not a token
that failed late).
281 tests pass; the same four as before do not.
|
|
Two traits over the same panel really do draw two different pictures --
verified on BXD.4, where all 20 chromosomes' polylines differ and the peaks
are LOD 2.93 at chr8 69.4 Mb (mCV24409492) against LOD 4.78 at chr8
95.7 Mb (rs3666069) -- but you could not tell from the title: every plot
came out headed **"trait rqtl: LOD along the genome"**, because
`/plot/rqtl` labelled it from `?trait=` and nothing else, defaulting to
"rqtl". The only visible difference was the peak annotation, and a caller
posting the saved response back (which is the documented way) could not
know the label was missing.
The response now names what it mapped: `/compute/rqtl` writes `Trait:
<name>` as its **first** note, and `/plot/rqtl` and `/plot/gemma` take the
title's label from that note when `?trait=` was not given. The
`POST /plot/rqtl` suggestion in `actions` also carries `?trait=`, so a
caller that posts only `data` still gets the right title.
* src/routes/rqtl.rs: the `Trait:` note, ahead of the column note; the
plot suggestion's href names the trait.
* src/routes/plot.rs: `trait_from_document`, used by both plot endpoints.
* tests/plot.rs: `two_traits_draw_two_different_plots` -- two canned
responses differing only in LOD, asserting each title names its own
trait from its own note, each annotates its own peak and not the
other's, and every polyline differs. That is the regression this is
about: a plot that reused anything between requests would draw the
first trait's shape for the second.
* tests/{rqtl,phewas}.rs: the plot suggestion is now matched by prefix,
with the trait asserted.
* README.md, docs/www/ai-instruction.md: where the title comes from, and
to post the whole response (or add `?trait=`) if you post only `data`.
|
|
One threshold for both engines was the wrong shape. For R/qtl the
interesting line is where a `scanone` stops being seconds (a few hundred
individuals is seconds of one core), so 1 000 stays. For GEMMA it is much
earlier, and not because of the runtime:
- every GEMMA run goes through **one shared cache directory**
(`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome
kinship archives of hundreds of megabytes by content hash. Two runs at
once are two writers over the same archives, and a half-extracted kinship
another run reads surfaces as a GEMMA failure;
- a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382
individuals write 207 MB of kinship per chromosome.
So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every
mapping request is gated in practice, and only a handful of individuals (a
four-way cross's parents, a probe of the pipeline) runs free. `ravanan`
uses the same number, since it writes into the same cache. The threshold
moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint
carries its own; the policy, the slots and the 409 are unchanged.
Tests, asked and answered: at 50 the three gemma tests that do real work
(fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated
and share one token, so on a cold cache two of them at once would collide
into `409 compute_busy` -- a gate doing its job, failing a test that only
wanted to map something. They now take a lock, and only they do; the
validation tests stay parallel, and the rqtl suite is untouched (its gate
still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an
empty directory, `cargo test --test gemma` is green twice in a row, 18 s
per run, i.e. really recomputing.
* src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL`
(50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`;
the test that pinned one number now pins both and that each endpoint is
wired to its own.
* tests/gemma.rs: the three real-job tests take the lock, and say why.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
the two thresholds and the cache-directory reason for the lower one.
277 tests pass; the same four as before do not.
|
|
Two things that came out of reading a real response rather than a test.
**Say what a case attribute means.** `GET /groups/{group}/case-attributes`
returned names and values, and the per-trait attribute endpoint returned
values, so a caller had to guess that `SeqCvge` is a sequencing depth.
MariaDB has had the answer all along in `CaseAttribute.Description`:
SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016
Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct
Epoch BXD family subgroups. Each number with common parents. ...
Sex F= female, M= male
Both endpoints now carry it -- a `descriptions` map on the group listing, a
`description` field on the per-trait one -- and the covariate notes quote it,
so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for
1 male of each strain in 2016): categorical ..." instead of leaving the name
to be interpreted. It matters most for covariates: `SeqCvge` is a property
of the *genotype* data, its values include compound ones (`40+31L`, two
sequencing runs), and the server codes those as 20-odd levels rather than as
a depth -- a covariate fitted for the wrong reason, which the description is
the first hint against.
**Gate large computations, not every computation.** The gate was applied to
every job, so a 93-strain test request could be refused because somebody
else's 6 000-individual run held the slot -- which is how the test suite
started serialising itself behind a mutex, a workaround for the wrong
problem. What the gate is for is *large* work: a BXD panel of ~200 finishes
in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB
of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000
individuals): below that no slot is taken, nothing waits and nothing is
refused; above it the old policy holds unchanged (1 slot for gemma, 1 for
ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409
naming the run in the way).
The size is not known when the request arrives -- the panel is resolved
server-side -- so the decorator now makes a *light* claim and the handler
promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the
point where it knows, then hands it to `Gate::run` around the call that
computes, so the slot still outlives a client disconnect. A light claim
registers nothing and its drop removes nothing: only a held claim may touch
the claimant map.
* src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`,
`Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved
out of `claim_or_busy`), and `Gate::run` unchanged in spirit.
* gnauth-macros: binds the light claim; the body keeps one gate call.
* src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is
known, and the compute call wrapped in `run`; error mapping moved inside
the job so the future returns `ApiError`.
* src/rqtl.rs: `Sample` is `Clone` (the job owns its copy).
* tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are
under the threshold); rqtl keeps its lock only for the machine's sake --
concurrent `Rscript` scans would compete for cores and scratch, not for a
slot. Four new gate tests pin the threshold, that a small job never takes
the slot, that one caller cannot hold two large jobs, and that another
caller waits rather than fails.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
what the threshold is and why it is 1 000.
Measured on :3001 after the change: two concurrent small-panel GEMMA
requests from one token both answer 200 (one was 409 before), while two
6 382-individual requests answer 502/409 -- the second gated with the usual
`compute_busy` naming the first.
277 tests pass; the same four as before do not.
|
|
The shared covariate note promised "5 dummy column(s) named Status.<level>"
on both compute endpoints, but only one of them gets dummies: GEMMA is
handed k-1 columns and R/qtl's wrapper is handed the single level column
and expands it itself. On `/compute/rqtl` the sentence described something
that never appears -- caught by reading a live response on production,
where the note said "5 dummy column(s) named Status." one line above the
Rscript command line that passes no such thing.
The note is now engine-neutral -- `... 90 of 93 samples have a value, 34
with both the trait and the covariate; 6 levels` -- and GEMMA says what it
was handed in its own note, which is the part a reader comparing two runs
wants: `GEMMA fits 4 covariate column(s): Status.Live, Status.UTHSC,
Status.UTHSC2, ...`. The unit test asserts the level count, and the
end-to-end test asserts the columns note names `Status.` columns twice or
more, so a covariate silently fitted as one ordinal number fails it.
* src/covariates.rs (resolve), src/gemma.rs (compute), tests/gemma.rs,
docs/www/ai-instruction.md (the example note).
|
|
`gemma-wrapper --permutate n` repeats the *whole* LOCO association scan
per permutation -- it shuffles the phenotype file and starts over -- so a
thousand permutations is thousands of full scans: hours to days on a real
panel, against the single slot `/compute/gemma` has. That is not a test
this deployment carries, and an assistant asked for significance
thresholds will ask for it, so the answer has to be an explicit one.
Until now the request type simply had no such field and serde ignores
unknown keys, which is the worst of the three outcomes: the caller gets a
200, no thresholds, and no indication that what it asked for was dropped.
`nperm`, `permutations` and `permutate` are now named fields and refused
with `422 permutations_not_supported`, naming the cost, pointing at
`POST /compute/rqtl` with `nperm` (the endpoint that does have the test,
over the same panel and values) and at the `actions` member every
successful GEMMA response already carries, pre-filled with exactly that
call. Zero keeps its usual meaning -- off -- so a client sharing one body
shape across the two endpoints still works.
* src/routes/gemma.rs: the three fields, the refusal before anything
expensive runs, and the response note now saying permutations are
refused rather than merely absent.
* tests/gemma.rs: `compute_gemma_refuses_permutations` pins all three
spellings, the code, and that `nperm: 0` is not a request for any. It
takes the compute lock: the gate answers before the handler does, so a
concurrent job in the same test binary came back 409 instead.
* README.md, docs/www/ai-instruction.md: what to do instead -- judge a
peak against the smallest `p_lrt`, and get thresholds from R/qtl.
273 tests pass; the same four as before do not.
|