| Age | Commit message (Collapse) | Author |
|
The gotcha notes in SKILL.md tell an agent to count the animals per
genotype class before believing a peak; a note in a document is only
found by whoever reads it, so the replies now say it too, at the peak
they just computed.
* suggest::class_hint builds
/groups/{g}/genotypes/{v}/markers/{peak}/classes?trait={trait_id} --
a real GET with one placeholder, because a scan knows the trait's
values, not its PublishXRef.Id.
* /compute/{rqtl,gemma,qtl2,fitqtl} add it to `next` after the PheWAS
link, naming the peak each reply found and the version the request
named (a {version} placeholder when it named none).
* The genotype subtree carries the template too: a version's detail,
and a concrete marker's own links.
* /compute/qtl2 gains the sentence that explains the divergence: on a
riself (riset) panel heterozygous calls are dropped before the scan
(R/qtl's convert2riself) and the HMM re-imputes them, while GEMMA
keeps them at dosage 0.5, so a mostly-H marker cannot peak there and
two engines can name different markers of one region without either
being wrong.
tests/links.rs: a scan's reply offers the class counts at its own peak --
the link names the version and the peak marker the reply reports, and
leaves {trait_id} for the caller.
|
|
SKILL.md told an agent how to call the endpoints, not how to read them
when two engines disagree -- which is the question the last round of
analysis actually asked. A note in "Gotchas that cost time":
* Count the animals in the peak's genotype class before believing it
(.../markers/{marker}/classes?trait=<id>). DOL 10003's GEMMA peak is
A n=2 mean 1.8 against B n=171 mean 3.95, with its next two hits the
same pattern mirrored: a haplotype carried by a handful of extreme
animals, not a population effect.
* H is dropped for R/qtl2 on a riself panel (kept by GEMMA as dosage
0.5); its HMM re-imputes, so the observation is lost, not the animal.
A mostly-H marker becomes nearly monomorphic for R/qtl2 -- 2 A vs 171
B -- which is why the two engines can name different markers of one
region. Neither is wrong.
* GEMMA's af is mean(dosage)/2 under B=1, H=0.5, A=0, not an allele
frequency (0.469 is half of 0.935); allele frequencies come from
.../markers/{marker}/frequencies, and allele1/allele0 are the BIMBAM
X, Y placeholders, so the sign of beta is not a strain.
* Do not compare scores across engines as one scale: GEMMA -log10 p,
R/qtl/rqtl2 LOD, qtlreaper LRS (lod = lrs / 4.604).
|
|
The question that follows "why do GEMMA and R/qtl2 peak at different
markers": what does the trait look like within each genotype class at one
marker. The frequencies endpoints report the distribution of calls, not
the values behind them, and the answer must not be "serve the matrix" --
this API never does.
GET /groups/{group}/genotypes/{version}/markers/{marker}/classes?trait=<id>
-> {"A": {"n", "mean", "sd", "min", "max"}, "H": {...}, "B": {...}}
The calls are joined to the trait's per-strain values inside the server
(the same lookup, authorization and 404s as /groups/{g}/traits/{id}/values)
and reduced to those five numbers per class. Nothing in the payload names
an animal. A class with fewer than ten animals is called out in notes,
because a peak resting on two is a different claim from one resting on a
hundred.
It settles the DOL trait 10003 question. At GEMMA's top marker
JAX00418132 (chr16 36.35 Mb) the classes are A n=2 mean 1.8, H n=20 mean
2.53, B n=171 mean 3.95 -- the A class is two animals and they are the
trait's two lowest values. R/qtl2 drops H on a riself panel (R/qtl's
convert2riself), so it keeps 173 animals there but only 2 of one class:
nearly monomorphic, which is why its scan peaks elsewhere. GEMMA keeps
the 20 hets at dosage 0.5, and that is where its sharper p-value comes
from. GEMMA's next two hits are the same pattern in opposite directions
(one A among 171, then two B among 171), so the chr16 block is a haplotype
carried by a handful of extreme animals, not a population effect.
Also fixes the reading of GEMMA's `af`: it is mean(dosage)/2 under the
BIMBAM coding this deployment ships (B=1, H=0.5, A=0), so 0.469 for
JAX00418132 is half the B-allele frequency 0.935, not the A-allele
frequency 0.065. The bimbam file and DOL.geno agree at that marker.
tests/frequencies.rs gains a route test: the class fields, A < B, the
small-class note, the "not served" note, no individual names anywhere in
the payload, and a 404 for a trait the group does not have.
|
|
A GeneNetwork trait page shows the gene's aliases next to the symbol,
and GN4 had no way to ask for them: gn-guile has a Wikidata lookup
(gn/db/sources/wikidata.scm) and nothing in GN4 did. This is that
lookup with the caching Wikidata deserves -- it is a shared public
service and an alias list is not news.
* GET /genes/{symbol}/aliases: two SPARQL queries against
WIKIDATA_ENDPOINT (the entity: a gene wdt:P31 wd:Q7187 in human,
mouse or rat, labelled with the symbol; then its skos:altLabel s in
English), cached in WIKIDATA_CACHE_DIR/gene-aliases.lmdb, an LMDB
file (NO_SUB_DIR, 0600, so one file and LMDB's -lock companion
rather than a directory of two).
* TTL WIKIDATA_TTL_SECONDS, seven days by default. Past it the value
is refetched; ?refresh=true forces that.
* The label is matched case-insensitively, which the gn-guile query
did not: human genes are labelled SHH and mouse ones Shh, so an
exact match finds only one species. ?species=human now returns the
ortholog's labels -- HLP3, HPE3 and the rest -- which are the ones a
mouse trait page shows. With no species named the pick is
deterministic (mouse, rat, human), so a repeated request cannot
change entity.
* A failed refresh keeps the last value: the answer says stale: true,
the envelope is partial, and a note explains it. Nothing cached at
all is a 502 that says so. An empty alias list and an unreachable
upstream are different facts, and only one of them is about the
gene.
Seven unit tests (key scoping, TTL including a clock that went backwards,
the query shape and quote-escaping, SPARQL flattening, an LMDB
round-trip that asserts the file is gene-aliases.lmdb, and
stale-on-failure with and without a cache) and two route tests (the
second call is served from the cache -- same fetched_at -- and the
species switch), which skip with a 502 assertion when there is no
network.
|
|
A probeset id is not a Xapian field, so /search cannot find 1427571_at
exactly and the only route to it was guessing a dataset and calling
/datasets/{ds}/probesets/{name} until one answered. A gene symbol could
be searched, but the reply was capped at 100 hits, interleaved probesets
and platforms, and carried no total -- so "did I see all of Shh?" had no
answer. Both questions are one indexed query each: ProbeSet.Name_IDX and
symbol_IDX.
* GET /probesets/{name}: one row per dataset that carries the probeset,
with the panel (group, species, platform) and the scan's statistics
(mean, LRS, LOD, locus, additive). Suffixed chip variants match too
-- 1427571_at also finds 1427571_at_A -- and each row's `name` says
which matched, so the answer never hides a substitution. LIKE's
wildcards are escaped, or every Affymetrix `_at` would match any
character. 155 rows over 155 datasets in 32 ms.
* GET /genes/{symbol}/traits: the same rows filtered on Symbol, with a
real total, ?sort=lod, ?min_lod=, ?group=, ?species=, ?platform= and
paging -- "BXD datasets where Shh is a strong eQTL" is one call.
1225 rows over 17 datasets and 13 probesets in 38 ms.
* Both serve metadata, not values. A token is enough for the index;
trait values stay per-dataset and authorized per dataset, and `next`
goes to /datasets/{ds}/probesets/{name}/values, which checks. There
is no cross-dataset values endpoint, deliberately: that is bulk work
and /groups/{g}/traits is the bulk call.
The one-statement join took 5.2 s for Shh -- MySQL rescans
InbredSet/Species per row -- so the 933 ProbeSetFreeze rows and the 102
panels are fetched separately and joined in memory: 70 ms each. nsamples
is left out: it is a ProbeSetData count per row (+0.5 s) and the dataset
record already reports strains.
|
|
A caller that knows only /version should be able to reach anything by
following links. Until now `next` was thin and uneven: the genotype
subtree offered none at all, the group record offered one, /version
offered nothing, and two links it did offer could not be followed.
* suggest:: now builds the graph: root_links (groups, species,
mapping-methods, openapi), genotype_version_links (frequencies,
marker names, a marker's location and tally, the group, the scanning
actions), unlisted_genotype_links, marker_links (location, tally,
marker list, panel tally, PheWAS at that marker), group_links and
trait_links. Envelope::with_links / with_links_except add them,
deduplicating and never suggesting the response the caller is
already reading.
* Wired through the genotype surface (listing, version detail, markers,
location), both frequencies routes, the compute endpoints, groups,
traits, both value routes, species, datasets, publications,
accessions, search, mapping methods, phewas, correlation and
/version.
* Fixed what was not followable, because `next` promises a GET that
returns 200 and a client is the one that finds out otherwise:
- /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are
gone; comput endpoints now link ?locus=<their strongest marker>
(suggest::peak_marker_of, computed before the rows move).
- /datasets and /publications are not routes; root_links offers the
listings that exist (/groups, /species, /mapping-methods) and
/openapi.json.
- /search?q= with an empty query is a 400, so it is not a link.
- the correlation endpoint's "/compute/correlation" was a POST in
`next`; it is in `actions` now.
- marker_links pointed at /markers/{marker}, which is not a route;
it points at /markers/{marker}/location.
tests/links.rs walks the graph two hops from a dozen seeds and fails on
any link that does not answer 200, plus a check that a tally leads to the
marker list and to a marker's location, and a location to that marker's
tally and its PheWAS.
|
|
Two questions the genotype surface could not answer: what a panel is
actually called as, and which markers it holds. The listing gives counts
and the compute endpoints give scans; a panel whose calls are not the
codes its header declares, or a marker that is 99% missing, stayed
invisible until a scan came back flat -- and then the search for the
cause started at the wrong end.
* GET /groups/{group}/genotypes/{version}/frequencies counts the calls
in the version's .geno and returns them as call -> frequency
({"B": 0.4889, "D": 0.492109, "H": 0.018991}). Frequencies are over
the calls that are present, so they sum to 1; missing codes (U, N,
NA, an empty cell) are counted separately and reported in notes,
because the missing rate is the diagnosis.
* A probability-valued file is binned 0.05 wide (twenty bins over
[0, 1], keys are the bin edges), since 0.4187 and 0.4191 are the same
statement about a marker; ?bin= sets another width. A file with a few
distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported
exactly -- binning three values hides what is being looked at.
* .../markers/{marker}/frequencies is one locus, and ?marker= on the
panel route is the same call. Counting goes through an interner, so
cost is one hash lookup per cell and one allocation per distinct
call; a .geno over 256 MB is refused (413 genotype_too_large) rather
than read, which is what HSNIH-Palmer_true (3.7 GB, 134927 markers x
6147 individuals) demanded.
* The version's /markers now lists marker NAMES only, one column, and
.../markers/{marker}/location returns {"name","chr","cM","Mb"} as
named fields: a page of names stays small, and a position is a
lookup rather than a table the client must hold. The same size guard
applies, so an oversized panel's listing is a 413 in 3 ms.
* Files no group lists -- a panel in preparation, a fixture, the DO's
do/ cross -- live at /groups/unlisted/genotypes/{name}/... with the
same four calls. `unlisted` is a namespace, not a group: only its
genotype subtree answers. One prefix for the whole genotype surface
beats a second spelling for the same tally, so the earlier
/genotypes/{name} spelling is gone.
* genotypes::resolve_named (moved out of the frequencies module, since
it is genotype metadata) resolves a version, a .geno name, or a path
relative to the genotype files directory; MAX_SCAN_BYTES is shared by
every whole-file read.
Tests limited to the feature: genotypes 3 (names-only columns, paging,
location fields, marker_not_found, unlisted parity, 413), frequencies 5,
frequencies unit 7.
|
|
A multiparent population cannot come from a .geno. That file carries two
codes per marker (B/D, A/H/B) -- a collapse, not a founder allele -- and
the collapse is not invertible, so DOL's 194 DO animals cannot be fitted
as a DO however much reference data appears. What can be is a cross that
ships its own qtl2 control file with founder genotypes, so the endpoint
now takes one by name:
{"rqtl2_cross": "do", "trait_name": "DO::10001 (WBC)", "samples": ...}
* resolve_control_file looks in genotype/rqtl2/{name}.json, then
{name}/{name}.json (do/do.json), then {name}.json under
GENOTYPE_FILES; prepare_control_cross reads the cross's pheno, maps
and founder_geno headers without converting anything. It reports
crosstype, founders and whether the cross is multiparent, and
refuses a multiparent crosstype with no founder_geno rather than
silently scanning it as biallelic.
* Samples are matched to the cross's own population by name and the
strays are counted in notes; rqtl2_cross cannot be combined with
group or genotype_file.
* The gate is given individuals x founders, because a multiparent scan
fits one coefficient per founder allele at every marker.
* Autosomes only now applies to shipped crosses as well: the DO's X is
left out (369 markers) instead of being fitted without a sex
covariate, and the script subsets with cross[, chr = autosomal].
Measured on the DO (WBC, 742 individuals, 8 founders, 6044 autosomal
markers): probabilities 174 s / 1.39 GB (cached), LOCO kinship 15 s
(cached), scan 21 s. scan1 runs on one core for a multiparent cross
because its forked path stalls on allele probabilities here -- 25+
minutes at 8 cores with no CPU consumed, against 21 s serial.
Two bugs found by running an actual published trait through it:
* do_pmap.csv is in Mb, not the bp qtl2 documents, so every DO peak was
reported at 0.0 Mb. The map's unit is now detected from its largest
position and printed.
* The phenotype file was read with the default whitespace separator, so
a label with a space ("DO::10001 (WBC)") split into three fields and
read.table stopped with "line 1 did not have 3 elements". Both the
pheno and covariate reads are tab-separated now.
The R error extraction also joins R's wrapped continuation line, so a
complaint reads as a sentence rather than "Error in scan(...) :".
|
|
R/qtl sits between the existing two scans: Haley-Knott on marker
genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with
mean-imputed genotypes. Neither can analyse a multiparent population,
and neither models relatedness from the actual genotype data. R/qtl2
does both -- HMM genotype probabilities, then an LMM with
leave-one-chromosome-out kinship -- and it is what names the third peak
on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24,
against GEMMA's -log10 p 7.15 there).
* src/qtl2.rs converts this deployment's own .geno into a qtl2 cross
under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with
the genotype-code map, the orientation and the paths written out
explicitly. The shipped genotype/rqtl2/ control files cannot be
used -- they point outside the data directory, declare no code map
(every genotype reads as missing) and no orientation (individuals
read as markers) -- and each of those produces a plausible flat
scan instead of an error, which is why the conversion is ours.
* Heterozygous calls are dropped on a riself panel, as R/qtl's
convert2riself does. Left in, qtl2 reads the cross as multiparent
and its scan1 takes a path the panel cannot satisfy ("Error in
scan1_pg ... : No individuals in common").
* Autosomes only: X needs a sex covariate to be fitted properly and
Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is
restricted the same way. The note says how many markers were left
out.
* scripts/qtl2_scan.R caches genotype probabilities and kinship by a
hash of the control file *and its directory*; two conversions of
the same panel can have identical control files and identical file
sizes, and reusing the first one's probabilities silently scans the
wrong cross. It also takes the individual names from whichever
shape the cross's pheno slot has -- a single-column pheno file
leaves it a plain matrix, and the resulting empty phenotype reads
as scan1's "No individuals in common".
* Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply
unchanged -- post the same samples as /compute/rqtl.
* tests/qtl2.rs: the fixture scans and its notes name the cross; BXD
trait 10001 peaks on chr8 under both scans; refusals carry
missing_genotype / invalid_kinship.
cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like
R/qtl; a DO cross costs minutes of genotype probabilities, paid once
per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
|
|
`/compute/fitqtl` has taken `covariates` since c642f87, and the skill
mentioned it only as a trailing sentence inside the fitqtl bullet -- after
the paragraph that says "both engines", which an assistant skimming for
covariates reads as rqtl-and-gemma. Three changes so the offer is where a
reader looks for it:
* the covariates bullet now names all three endpoints
(`/compute/rqtl`, `/compute/gemma` **and `/compute/fitqtl`**);
* the fitqtl bullet says it in bold, with why it matters (the `additive` is
then the effect *after* conditioning), and adds the consequence that the
LOD no longer matches the scan once a covariate is fitted -- the earlier
wording claimed parity unconditionally;
* the endpoint table gained `POST /compute/{rqtl,gemma,fitqtl}` (it had no
entry at all), and the session outline now includes sizing a peak with
fitqtl when the user asks how much it matters.
Docs only.
|
|
`fitqtl` takes a `covar` argument, so the effect it reports can be the one
*after* conditioning -- which is usually the number worth quoting for a peak
that overlaps a known confounder. `covariates` now works on /compute/fitqtl
exactly as on /compute/rqtl: the same `covariates::resolve` (case attribute
by name, inline values, explicit map, same coding and same note), so the two
endpoints report a covariate identically.
The R side does what R/qtl needs: the phenotype file carries one column per
covariate after the trait, `--covarstruct` says which are categorical, and
the script dummy-expands those (k-1 against the first sorted level, missing
values left missing). Two things it has to get right:
* `model.matrix()` drops rows with a missing value, which silently misaligns
the covariate frame against the cross -- so the dummies are built directly
(`ifelse(is.na(x), NA, x == level)`), not through `model.matrix`;
* a covariate can be perfectly spread across the panel and **constant among
the individuals a trait covers** -- BXD's `Status` is exactly that -- and
R/qtl then stops with `Lapack routine dgesv: system is exactly singular`.
Columns that do not vary among the phenotyped individuals are left out,
and the reply names them, the same rule the GEMMA covariate file applies.
Measured on BXD trait 10001 at its peak, `BXD.4`:
no covariate LOD 2.9310 additive +2.394 32.8% var
with Status LOD 3.3602 additive +2.463 36.6% var
with an inline numeric covariate LOD 2.9373 additive +2.361
The LOD is no longer the scan's once a covariate is fitted -- it is a
different model -- which the docs now say, next to the no-covariate parity
check that still holds.
* scripts/qtl_fitqtl.R: `--covarstruct`, dummy expansion, the degenerate
column rule, and the dropped names in the summary line.
* src/fitqtl.rs: writes the covariates into the phenotype file and the
struct file, and passes both.
* src/routes/fitqtl.rs: the `covariates` field, resolved after alignment so
the coding and the complete-case count match /compute/rqtl.
* tests/fitqtl.rs: a covariate run (the note, the fitted row, a changed LOD)
and an unknown attribute refused with the list of valid ones.
* README.md, docs/www/SKILL.md: covariates on this endpoint, and why its LOD
stops matching the scan's.
306 tests pass; the same four as before do not.
|
|
`/compute/rqtl` answers "where" -- scanone gives a LOD per marker -- but not
"how much": a scan statistic is not a fitted model. R/qtl's `fitqtl` fits
one, and reports the QTL's effect, its standard error, the model LOD and the
variance explained. That is what a table of effects needs, and it is a
separate endpoint because it is a separate question, asked about a handful
of markers (a scan's peaks) rather than a genome: `fitqtl` fits one model
per marker, so a genome-wide version would be thousands of fits. Cap: 100
markers per request.
Rows are positional:
marker, chr, pos, lod, pct_var, additive, se, t, p
`additive` is fitqtl's estimate for the QTL term -- R/qtl codes the QTL
genotype +-1, so it is half the difference between the two homozygote
classes, the same convention GeneNetwork's published `additive` uses.
**Two independent checks that the numbers are right.** On BXD trait 10001
(`BXD.4`), at the scan's peak:
* `lod` = 2.930985, and `/compute/rqtl`'s scanone gives 2.930985 at the same
marker -- identical to every digit the CSV carries, because they are the
same cross and the same phenotype. That parity is now an integration test;
* `additive` = 2.394472, against the value genenetwork.org publishes for that
trait (`/api/v_pre1/trait/BXD/10001` -> 2.394444), which is 28 ppm away --
and that published number came from qtlreaper, a different engine.
* scripts/qtl_fitqtl.R (new): reads the same `.geno` and two-column
phenotype file as `rqtl_wrapper.R` (which is a verbatim copy of GN3's
script and so not extended), builds the cross the same way, fits
`fitqtl` at each named marker with `get.ests=TRUE`, and writes one CSV
row per marker -- plus a one-line summary naming the strongest fit.
`calc.genoprob(step=0)`: marker-level probabilities are all a
marker-level fit needs.
* src/fitqtl.rs (new): runs it, parses the CSV into positional rows (`chr`
stays text, it can be `X`), keeps the command line for `notes`.
* src/routes/fitqtl.rs (new): the endpoint -- same alignment rules as
`/compute/rqtl` (by name onto the panel, parents/F1s excluded unless
asked), the same `normalize`/`winsor_*` pipeline (effects come out on
that scale), and the same R/qtl gate, so a fitqtl request and a scan
queue against one another rather than competing for cores.
* src/config.rs: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`).
* tests/fitqtl.rs: the fixture (columns, sign of the effect, notes), the
LOD parity against scanone, and the refusals (no markers, both genotype
inputs).
* README.md, docs/www/SKILL.md: the endpoint, what each column means, and
when to reach for it instead of the scan.
305 tests pass; the same four as before do not.
|
|
Fitting `SeqCvge` as a covariate came back as `502 gemma_failed` with a
message about LU decomposition: its values are depths (`40`, `41`) with a
few two-library entries (`36+39L`), so the attribute was coded as **twenty
categories** and GEMMA died on the resulting dummies. The failure named
GEMMA, which is the wrong place to look.
`code_case_attribute` now refuses that shape at the covariate step, naming
the values and what to send instead:
422 invalid_covariate: case attribute 'SeqCvge': the attribute mixes
numbers with number-like text (14 of 19 values look numeric, but e.g.
'36+30L' does not), so coding it as categories would fit a quantity as a
factor -- and GEMMA fails on the result. Pass an explicit `map` giving
each of those a number (there are 5 such value(s): ...), or send the
numbers inline.
The rule is deliberately narrow: an odd level with *no* digits (`1, 2,
unknown`) is still a level and still coded, so nothing that worked stops
working; only dirty numbers are caught.
Also, the skill's covariate coverage -- which the review above found thin:
* when a covariate is worth trying (metadata that could confound: sex, age,
epoch, batch, sequencing depth), and the covariate step in the session
outline;
* read `.data.descriptions` before choosing one -- they are the only place
that says `SeqCvge` is sequencing depth and `Epoch` a family subgroup --
and the endpoint table now says so;
* read the covariate line in `notes` for the coding and the complete-case
n, and report that n beside any LOD/p change; plus `Cyro`/`Cryo` being
two levels in BXD;
* the categorical-vs-numeric rule measured rather than asserted: `RRID`
refused (82 distinct), `Status`/`Epoch` accepted, and the compound-value
trap written up with the `map` workaround;
* GEMMA's dummy columns, the column it drops when a level has no fitted
individual, and that R/qtl's note has no such line; missing values
dropping individuals; the 8-covariate cap and the `group` requirement.
* src/covariates.rs: the refusal, with a unit test for both sides of it;
* tests/gemma.rs: `SeqCvge` is a 422 naming `map`, not a GEMMA failure;
* docs/www/SKILL.md: the covariate section, the session line, and the
case-attributes rows in the endpoint table.
300 tests pass; the same four as before do not.
|
|
GeneNetwork2's trait page offers a set of normalisations, and its mapping
page warns about outliers without ever fixing them. This adds both, under
one spelling that `/compute/rqtl`, `/compute/gemma` and
`/groups/{g}/traits/{id}/values` all share:
normalize none | log2 | log10 | sqrt | negate | zscore |
robust_zscore | rank (GN2's names `invert` and
`qnorm` accepted)
winsor_quantile clip each tail at a fraction, e.g. 0.05
winsor_fences clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule
They run in a fixed, reported order -- transform, then winsorise, then
standardise -- so clipping acts on the scale being analysed and the result
lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The
note gives the steps *in the order they ran* (`winsorized at Tukey fences,
then zscore`), the count clipped in each tail, and how many values sit
outside Tukey fences -- the warning GN2 gives before mapping and this API
gave nowhere.
**Checked against genenetwork.org itself.** GN2's public trait page embeds
its own reference values (`js_data.qnorm_values` / `zscore_values`, the
`data-qnorm`/`data-zscore` attributes), so there is an oracle for the two
transforms whose definition matters. On BXD trait 10010 (28 samples):
`rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's
three-decimal rounding. Getting there found a real disagreement: GN2 uses
`scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use
the **population** sd; ours used the sample sd and came out
`sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2.
Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based
inverse-normal (Blom) transform, so it is called `rank` here; and its
`sqrt` silently adds 1 when a value is in [0,1], which shifts every effect
size -- a negative value is refused instead.
* src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom
scores over an Acklam inverse-normal, `Params` for query/body use,
`to_json` for suggestions, and eleven unit tests -- the quantile table,
ties sharing an average rank, missing values never imputed, GN2's own
`find_outliers` example, and the note's wording.
* src/routes/{rqtl,gemma}.rs: the three fields, applied **after**
alignment (over the samples that enter the fit) and before anything reads
the values, with the note; the R/qtl permutation suggestion and GEMMA's
R/qtl suggestion carry the pipeline.
* src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values
endpoint, and the mapping bodies in `actions` carry the same fields -- a
caller who fetches z-scores must map those numbers, not the raw ones.
* tests/groups.rs: values are mean 0 / population sd 1, the note says so,
every `/compute/*` suggestion in `actions` carries it, and an impossible
pipeline is a 422 `invalid_normalization`.
* README.md, docs/www/SKILL.md: what each name means, the order, the
population-sd parity, and that effects come out on the transformed scale.
298 tests pass; the same four as before do not.
|
|
A GEMMA row now carries the effect beside the p-value it belongs to:
`… beta, se, l_mle, additive, p_lrt`. `additive` is `-beta/2`, which is
what GeneNetwork's own tables and CSV call the additive effect
(gn2/wqflask/marker_regression/gemma_mapping.py:240). Halved because half
the difference between the two homozygotes is the usual additive effect;
negated because GN2's BIMBAM conversion codes `@mat` as 1, so GEMMA's beta
is per *maternal* copy while GeneNetwork's published sign is anchored on the
other allele. Two facts that make this worth doing carefully:
* `beta` and `se` are left exactly where GEMMA put them. `se` is what makes
the effect interpretable (`beta +/- 1.96*se` is a 95% interval) and both
are GEMMA's own numbers, so replacing them with a derived one would lose
information;
* the derived column goes *before* the p-value, not after it. `p_lrt` must
stay the last column: `gemma::peak_marker` reads `row.last()` for the peak
(and with it the PheWAS and plot suggestions), and the plotters and the
docs all say "the last column is the p-value". Appending would have
silently pointed every peak at an effect size.
The response also says what the three columns mean, and that they describe
*this run* on these samples and this genotype version -- not the `additive`
trait info reports, which came from the published mapping (trait 20010:
this run -0.54, published +2.118, same peak marker).
Nothing is added when GEMMA wrote no `beta` (a non-LMM run): the columns
are then exactly GEMMA's.
* src/gemma.rs: `with_effect_size`, applied by `parse_assoc_file`, so
/compute/gemma and /compute/gemma/ravanan both get it.
* src/routes/gemma.rs: the note.
* tests: four unit tests (placement, sign and halving, `peak_marker` still
finds the peak from the last column, a table without beta untouched, a
null effect for a row with `NA` beta) and one end-to-end test asserting
the column sits immediately before `p_lrt`, every value is `-beta/2`,
`p_lrt` stays a probability, and the note is there.
* README.md (the response example and what the column means),
docs/www/SKILL.md (how to read and quote it).
286 tests pass; the same four as before do not.
|
|
Two GEMMA runs over one BXD panel drew genuinely different pictures --
traces differ on all 20 chromosomes, peaks -log10 p 5.48 at chr8 97.0 Mb
against 5.43 at chr1 175.6 Mb -- and both came out headed
**"trait gemma: GEMMA association along the genome"**, because GEMMA's
request had no trait label at all: `/compute/gemma` fell back to
`<group>::gemma`, and `/plot/gemma` fell back to "gemma". The plot of each
trait was told apart by its annotation and nothing else.
`trait_name` is now an optional field on `GemmaRequest`, used for the
`Trait: ...` note (first in `notes`, as on `/compute/rqtl`), for the PheWAS
and R/qtl suggestions, and for the `/plot/gemma?trait=` suggestion -- so a
posted response titles its own plot, and `?trait=` still overrides.
Without it nothing changes: the label stays `<group>::gemma`.
* src/routes/gemma.rs: the field, the label resolution, the note, the
plot href.
* tests/gemma.rs: the covariate run names its trait and asserts the note is
first; tests/plot.rs: a GEMMA plot titles itself from the note, and an
explicit `?trait=` wins over it.
* README.md (`/compute/gemma`, and the plot section), docs/www/SKILL.md
(the GEMMA options list, and what the plot title comes from).
281 tests pass; the same four as before do not.
|
|
|
|
|
|
|
|
Answering "is robw using this successfully, and the others?" from the
access logs took an hour of inference -- user agents, search terms, which
traits were mapped -- because nginx records the address and the subject is
only in the token. Two callers can share an address (an agent and a person
behind one NAT), one caller can move between addresses (an LLM's egress is
a /24), and the answer for a given person was "probably 80.115.1.132".
The API now sets `x-gn4-user: <token sub>` on every **authenticated**
response, and the `gn4` log format records it as `user=`. The subject is
only known inside the handler (the `AuthToken` extractor verifies the
token there), so a middleware opens a slot in the request extensions, the
extractor fills it, and the header is set on the way out; a request with no
valid token gets no header, which is the honest answer and reads as
`user=-`. The caller only ever sees an identity it already presented.
* src/auth.rs: `USER_HEADER`, `CallerSlot`, and the extractor recording
its subject.
* src/lib.rs: the `tag_caller` middleware, outermost so the slot exists
before any extractor runs.
* tests/auth.rs: `an_authenticated_response_names_its_caller` -- the header
equals the token's `sub`, and `/version` (no identity) carries none.
* etc/nginx-vesuvio.pu.ac.ke.conf (and the deployed copy of it): `user=`
in the log format.
* README.md, docs/nginx.md (the format, an `awk` one-liner for per-user
counts, and the caveat that old and new lines differ in field count),
docs/www/operator-checklist.md (`user=-` means no token, not a token
that failed late).
281 tests pass; the same four as before do not.
|
|
Two traits over the same panel really do draw two different pictures --
verified on BXD.4, where all 20 chromosomes' polylines differ and the peaks
are LOD 2.93 at chr8 69.4 Mb (mCV24409492) against LOD 4.78 at chr8
95.7 Mb (rs3666069) -- but you could not tell from the title: every plot
came out headed **"trait rqtl: LOD along the genome"**, because
`/plot/rqtl` labelled it from `?trait=` and nothing else, defaulting to
"rqtl". The only visible difference was the peak annotation, and a caller
posting the saved response back (which is the documented way) could not
know the label was missing.
The response now names what it mapped: `/compute/rqtl` writes `Trait:
<name>` as its **first** note, and `/plot/rqtl` and `/plot/gemma` take the
title's label from that note when `?trait=` was not given. The
`POST /plot/rqtl` suggestion in `actions` also carries `?trait=`, so a
caller that posts only `data` still gets the right title.
* src/routes/rqtl.rs: the `Trait:` note, ahead of the column note; the
plot suggestion's href names the trait.
* src/routes/plot.rs: `trait_from_document`, used by both plot endpoints.
* tests/plot.rs: `two_traits_draw_two_different_plots` -- two canned
responses differing only in LOD, asserting each title names its own
trait from its own note, each annotates its own peak and not the
other's, and every polyline differs. That is the regression this is
about: a plot that reused anything between requests would draw the
first trait's shape for the second.
* tests/{rqtl,phewas}.rs: the plot suggestion is now matched by prefix,
with the trait asserted.
* README.md, docs/www/ai-instruction.md: where the title comes from, and
to post the whole response (or add `?trait=`) if you post only `data`.
|
|
One threshold for both engines was the wrong shape. For R/qtl the
interesting line is where a `scanone` stops being seconds (a few hundred
individuals is seconds of one core), so 1 000 stays. For GEMMA it is much
earlier, and not because of the runtime:
- every GEMMA run goes through **one shared cache directory**
(`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome
kinship archives of hundreds of megabytes by content hash. Two runs at
once are two writers over the same archives, and a half-extracted kinship
another run reads surfaces as a GEMMA failure;
- a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382
individuals write 207 MB of kinship per chromosome.
So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every
mapping request is gated in practice, and only a handful of individuals (a
four-way cross's parents, a probe of the pipeline) runs free. `ravanan`
uses the same number, since it writes into the same cache. The threshold
moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint
carries its own; the policy, the slots and the 409 are unchanged.
Tests, asked and answered: at 50 the three gemma tests that do real work
(fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated
and share one token, so on a cold cache two of them at once would collide
into `409 compute_busy` -- a gate doing its job, failing a test that only
wanted to map something. They now take a lock, and only they do; the
validation tests stay parallel, and the rqtl suite is untouched (its gate
still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an
empty directory, `cargo test --test gemma` is green twice in a row, 18 s
per run, i.e. really recomputing.
* src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL`
(50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`;
the test that pinned one number now pins both and that each endpoint is
wired to its own.
* tests/gemma.rs: the three real-job tests take the lock, and say why.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
the two thresholds and the cache-directory reason for the lower one.
277 tests pass; the same four as before do not.
|
|
Two things that came out of reading a real response rather than a test.
**Say what a case attribute means.** `GET /groups/{group}/case-attributes`
returned names and values, and the per-trait attribute endpoint returned
values, so a caller had to guess that `SeqCvge` is a sequencing depth.
MariaDB has had the answer all along in `CaseAttribute.Description`:
SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016
Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct
Epoch BXD family subgroups. Each number with common parents. ...
Sex F= female, M= male
Both endpoints now carry it -- a `descriptions` map on the group listing, a
`description` field on the per-trait one -- and the covariate notes quote it,
so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for
1 male of each strain in 2016): categorical ..." instead of leaving the name
to be interpreted. It matters most for covariates: `SeqCvge` is a property
of the *genotype* data, its values include compound ones (`40+31L`, two
sequencing runs), and the server codes those as 20-odd levels rather than as
a depth -- a covariate fitted for the wrong reason, which the description is
the first hint against.
**Gate large computations, not every computation.** The gate was applied to
every job, so a 93-strain test request could be refused because somebody
else's 6 000-individual run held the slot -- which is how the test suite
started serialising itself behind a mutex, a workaround for the wrong
problem. What the gate is for is *large* work: a BXD panel of ~200 finishes
in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB
of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000
individuals): below that no slot is taken, nothing waits and nothing is
refused; above it the old policy holds unchanged (1 slot for gemma, 1 for
ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409
naming the run in the way).
The size is not known when the request arrives -- the panel is resolved
server-side -- so the decorator now makes a *light* claim and the handler
promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the
point where it knows, then hands it to `Gate::run` around the call that
computes, so the slot still outlives a client disconnect. A light claim
registers nothing and its drop removes nothing: only a held claim may touch
the claimant map.
* src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`,
`Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved
out of `claim_or_busy`), and `Gate::run` unchanged in spirit.
* gnauth-macros: binds the light claim; the body keeps one gate call.
* src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is
known, and the compute call wrapped in `run`; error mapping moved inside
the job so the future returns `ApiError`.
* src/rqtl.rs: `Sample` is `Clone` (the job owns its copy).
* tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are
under the threshold); rqtl keeps its lock only for the machine's sake --
concurrent `Rscript` scans would compete for cores and scratch, not for a
slot. Four new gate tests pin the threshold, that a small job never takes
the slot, that one caller cannot hold two large jobs, and that another
caller waits rather than fails.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
what the threshold is and why it is 1 000.
Measured on :3001 after the change: two concurrent small-panel GEMMA
requests from one token both answer 200 (one was 409 before), while two
6 382-individual requests answer 502/409 -- the second gated with the usual
`compute_busy` naming the first.
277 tests pass; the same four as before do not.
|
|
The restart section added a moment ago pointed at the trap and then walked
into its other half: `nohup env … ./target/release/genenetwork4 &` leaves
an `env` wrapper whose command line also contains the binary path, so
matching `*release/genenetwork4*` finds the wrapper -- the pid file named
a process that exits while the server keeps running, and the next
`kill $(cat /tmp/gn4.pid)` did nothing at all.
Read `argv[0]` (the first NUL-terminated word) and compare it to the
binary path instead; that finds the serving process and only it. Applied
here, it is what finally moved production off the previous build.
|
|
Deploying today, `nohup env … ./target/release/genenetwork4 &` left an
intermediate process's pid in `$!`, so `kill $(cat /tmp/gn4.pid)` stopped
that and the server kept answering. The replacement died with `AddrInUse`
-- in its own log, where nobody was looking -- and `/version` still
reported the previous commit. Everything looked fine: the port answered,
the smoke tests passed, the notes were the old ones.
The checklist now says what actually decides it: read `/proc` for the
process running the release binary (there is no `ps` on this box), and
believe `/version`'s `git` over a successful-looking start -- a `git` older
than the build just made is this mistake, not a build problem. Start the
replacement only once `:3000` has stopped answering, and write the
resolved pid into the pid file.
|
|
The shared covariate note promised "5 dummy column(s) named Status.<level>"
on both compute endpoints, but only one of them gets dummies: GEMMA is
handed k-1 columns and R/qtl's wrapper is handed the single level column
and expands it itself. On `/compute/rqtl` the sentence described something
that never appears -- caught by reading a live response on production,
where the note said "5 dummy column(s) named Status." one line above the
Rscript command line that passes no such thing.
The note is now engine-neutral -- `... 90 of 93 samples have a value, 34
with both the trait and the covariate; 6 levels` -- and GEMMA says what it
was handed in its own note, which is the part a reader comparing two runs
wants: `GEMMA fits 4 covariate column(s): Status.Live, Status.UTHSC,
Status.UTHSC2, ...`. The unit test asserts the level count, and the
end-to-end test asserts the columns note names `Status.` columns twice or
more, so a covariate silently fitted as one ordinal number fails it.
* src/covariates.rs (resolve), src/gemma.rs (compute), tests/gemma.rs,
docs/www/ai-instruction.md (the example note).
|
|
`gemma-wrapper --permutate n` repeats the *whole* LOCO association scan
per permutation -- it shuffles the phenotype file and starts over -- so a
thousand permutations is thousands of full scans: hours to days on a real
panel, against the single slot `/compute/gemma` has. That is not a test
this deployment carries, and an assistant asked for significance
thresholds will ask for it, so the answer has to be an explicit one.
Until now the request type simply had no such field and serde ignores
unknown keys, which is the worst of the three outcomes: the caller gets a
200, no thresholds, and no indication that what it asked for was dropped.
`nperm`, `permutations` and `permutate` are now named fields and refused
with `422 permutations_not_supported`, naming the cost, pointing at
`POST /compute/rqtl` with `nperm` (the endpoint that does have the test,
over the same panel and values) and at the `actions` member every
successful GEMMA response already carries, pre-filled with exactly that
call. Zero keeps its usual meaning -- off -- so a client sharing one body
shape across the two endpoints still works.
* src/routes/gemma.rs: the three fields, the refusal before anything
expensive runs, and the response note now saying permutations are
refused rather than merely absent.
* tests/gemma.rs: `compute_gemma_refuses_permutations` pins all three
spellings, the code, and that `nperm: 0` is not a request for any. It
takes the compute lock: the gate answers before the handler does, so a
concurrent job in the same test binary came back 409 instead.
* README.md, docs/www/ai-instruction.md: what to do instead -- judge a
peak against the smallest `p_lrt`, and get thresholds from R/qtl.
273 tests pass; the same four as before do not.
|
|
moves
`/version` reports the revision the *server* is running, and this
deployment is updated in place, so an assistant that cached
`ai-instruction.md` can end up calling an API that has grown options its
copy never mentions -- by default, silently.
The instructions now say what to do about it, in the two places a reader
sees a revision: pull the repository and re-read the page when `git` is
not the revision the session started with, and the same in the `/version`
section where that field is explained. It matters for the last few
commits in particular: covariates in `/compute/rqtl` and `/compute/gemma`
were added, and a stale copy would never know to ask for them.
`docs/www/operator-checklist.md` gets the operator's half: nothing here
enforces it, so a release that adds endpoints or options should say so
where tokens are handed out, and `source_digest` -- not the commit alone
-- is what identifies a build in a bug report.
|
|
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl`
does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample
numbers -- and fits it with `-c` on the association step only. That needed
a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample
names, positional over the genotype panel, numbers only. GEMMA has no
factors, so a categorical covariate is dummy-expanded here into k-1
columns named after their levels (`Status.Live`, the first level being the
reference), where R/qtl's wrapper does that expansion for itself.
The spec shape, the alignment onto the panel and the text coding are now
one module, `src/covariates.rs`, called by both routes; `/compute/rqtl`
behaves as before except that its note also reports the complete cases
(`... 34 with both the trait and the covariate`), which README and the
instructions had been promising all along.
A missing covariate value is written `NA`, which GEMMA treats exactly as
it treats a missing phenotype -- that individual leaves the fit (measured:
one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's
`-9`, which GEMMA fits as the value minus nine.
Found on live data, not in the fixtures: a level that exists in the panel
but that none of the *fitted* individuals carries becomes an all-zero dummy
column, and GEMMA stops with `LUDecomp ... input domain error` rather than
dropping the term the way R would. Columns that do not vary among the
individuals with both a trait and a covariate are now left out, and the
response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a
covariate left with no columns runs without one, with a note saying so.
Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes
its file arguments, so a different covariate set is a different cache entry
while the kinship already computed is still reused. Covariates need a
`group`: individuals are aligned by name, which an explicit
`genotype_file`/`snps_file` request does not carry (`422
covariate_unavailable`), and an attribute the group does not define lists
the ones it has.
* src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now
`CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`,
`Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file`
(moved out of src/rqtl.rs) and `write_gemma_covariates_file`.
* src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate
file, `-c` after `-a` and before `-lmm`, and the notes about dropped
individuals and dropped columns.
* src/routes/gemma.rs: the `covariates` field, resolved after the trait
values so the note can count complete cases; `RqtlCovariate` replaced by
the shared spec.
* tests: six new -- the GEMMA file (numerical with `NA`; two and three
levels; inline codes; a dropped degenerate column) and an end-to-end
`compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with
`Status`, which also pins that `-c` is in the GWA command and not in the
kinship one, and that the covariate changes the fit rather than reusing
the plain run's cache. The gemma tests now take a lock: the gate answers
a second job from the same caller with 409 instead of queueing it.
* README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl
points at the shared shape and explains `x` vs `NA`), docs/www/
ai-instruction.md (both endpoints; what differs in `notes`).
Also in the instructions: `/version` reporting a `git` value you have not
seen before now says explicitly to pull the repository and re-read
`docs/www/ai-instruction.md` before the next call. This deployment is
updated in place and a stale page does not announce itself -- the API
accepts the request without the new option, or answers with fields the old
text never mentions.
272 tests pass; the four that do not (benchmark, no_db, and two search
parity/proxy replays) fail identically at the parent commit.
|
|
`POST /compute/gemma/ravanan` is now gated like the other two: one
workflow at a time, one per caller, `409 compute_busy` naming the run and
linking `/compute/gemma/status/{id}` (ravanan has no status route of its
own). One slot because ravanan already fans its per-chromosome steps out
-- over Slurm, when that backend is selected -- so a second request would
only queue behind the same scratch directory. Its gate is separate from
`/compute/gemma`'s: different runners, different cache directories, so one
of each at once is the overlap `/compute/rqtl` is already allowed.
Verified with a stub `CCWL_CMD` that sleeps: the holder's request held the
slot for the full 5 s, the same caller's second request was refused at
once, and another caller's waited for the slot and then ran.
While testing that, `GemmaError::Io` turned out to print "failed to run
gemma-wrapper" for *any* I/O failure -- a missing `CCWL_CMD` produced
"failed to run gemma-wrapper: No such file or directory", which sent me
looking at the wrong binary. Spawn failures are now their own variant
naming the command, and plain file I/O says "I/O error: …".
* src/compute_gate.rs (ravanan, permits): `Gate::new(1)`; `permits()`
exposes the slot count, and a test pins all three endpoints' counts
(gemma 1, ravanan 1, rqtl RQTL_MAX_CONCURRENT).
* src/routes/gemma.rs: the decorator on compute_gemma_ravanan, with
`status = "/compute/gemma/status"`.
* src/gemma.rs (Spawn): names the command; the four run_streaming call
sites use it; unit test for both messages.
* README.md (/compute/gemma/ravanan), docs/www/ai-instruction.md (the
slot list is now 1 / 1 / 8), docs/www/operator-checklist.md: concurrency.
|
|
`POST /compute/rqtl` now takes up to `RQTL_MAX_CONCURRENT` slots (default
8, read when the gate is built) with the same one-job-per-caller rule as
GEMMA: a second submission from the same token while its scan is running
or queued is refused `409 compute_busy`; a submission that finds every
slot taken waits for one to come free. Eight because an R/qtl scan is a
single-threaded Rscript on a 96-core box; not more, because a scan on a
3 000-individual cross is minutes of CPU and the same machine serves the
database. Unlike GEMMA's single slot, concurrent scans share no cache
(R/qtl results are uncached), so the only cost is the CPU they take.
Two flaws only appeared once there was more than one slot:
- the "what is running" record was a single `Option<Active>`, so a
refusal quoted whichever job had most recently taken a slot -- robw's
409 named JoM's run id. It is now a map keyed by subject, and
`a_refusal_quotes_the_callers_own_run_id` pins it;
- `Busy::describe` took the endpoint identifier, so the message read
"your rqtl job"; the decorator takes a separate `label` ("R/qtl") from
the status path.
The test suite hit the rule itself: six compute tests in tests/rqtl.rs
share one token, and cargo runs them on several threads, so three were
refused instead of run. They now serialise on a module mutex, with a
comment saying why -- the same lesson clients need, that parallelism
means more tokens, not more requests from one.
* src/compute_gate.rs (rqtl()): eight slots from `RQTL_MAX_CONCURRENT`;
`Gate::new(permits)` was already parameterised. `running` becomes a map
keyed by subject, since one record cannot name N jobs. Module doc talks
about slots rather than a single slot. Tests: eight subjects run and the
ninth waits, a caller still may not take two even with slots free, and a
refusal quotes the caller's own run id.
* src/config.rs: RQTL_MAX_CONCURRENT, default 8, fixed at first use.
* gnauth-macros/src/lib.rs: `label = "…"`, so prose and status path are
independent.
* src/routes/rqtl.rs: the decorator, and `_token` becomes `token`, which
it needs.
* tests/rqtl.rs: compute tests serialised.
* README.md (/compute/rqtl), docs/www/ai-instruction.md (4a),
docs/www/operator-checklist.md: the numbers per endpoint, and that more
parallelism means more tokens.
|
|
`POST /compute/gemma` now takes a single slot: one job runs at a time
across all callers, a second submission from the same token is refused
immediately with `409 compute_busy` (whatever it would have waited for,
its own job is already in the way), and a submission from another token
waits for the running job and then runs. Nothing else is gated -- ravanan
and /compute/rqtl are untouched.
The gate is one `tokio::sync::Semaphore(1)` (fair: waiters take the
permit in arrival order) plus a `Mutex<HashSet<String>>` of subjects
holding the slot or queued for it, in a per-process static. The claim is
moved into the spawned job task, so a client disconnect -- which does not
kill the child process -- cannot hand the slot to somebody else while the
CPU is still busy.
The endpoint carries none of that: `#[compute_gate(gemma, run_id =
params.id.as_deref())]` expands to a claim and a run, and requires the
handler to take `Ctx` and `AuthToken` (every compute endpoint is
token-only, and the token's sub is the key), so an endpoint that is not
token-gated cannot be decorated. The policy -- the 409's status, code,
wording and next link -- stays in plain Rust (`Gate::claim_or_busy`), so
the gate's own tests cover it; the attribute is decoration over two calls.
* src/compute_gate.rs: new. Gate, Claim (released on drop, including a
panic), Busy, claim/claim_or_busy/run, and four tests, two of them on a
four-thread runtime (the earlier ones ran single-threaded and so never
exercised the cross-thread hand-off; one now asserts at most one task
inside the slot across eight tasks).
* gnauth-macros/src/lib.rs: the `compute_gate` attribute, alongside the
`gnauth` markers it shares `syn`/`quote` with; arguments documented
(gate, endpoint, status, run_id) for the endpoints still to be gated.
* src/routes/gemma.rs: the decorator replaces 57 lines of claim and
spawn plumbing in compute_gemma.
* docs/www/ai-instruction.md (4a, 5): one job at a time per caller, and
that a `compute_busy` 409 means poll, not resubmit.
|
|
* README.md (Running with cargo): "This deployment runs two instances:
3000 (production) and 3001 (development)" -- the table of ports, binaries
and cache directories, both start-up invocations, and why the caches are
separate while the database, index and auth stub are shared. Also notes
that in-memory state is per instance, so the dev server is not a capacity
test of production.
* docs/www/ai-instruction.md:
- /version now described honestly: git (with -dirty), dirty,
source_digest, database reachable/unreachable -- and that
`unreachable` means stop and report, not retry in a loop.
- error table: 413 (plain text, not the JSON object: over the 64 MiB
body limit), 500 database_error with a connection-loss detail is
transient (retry once), and 409 run_id_in_use pointing at the status
endpoint.
- covariates: case_attribute is the smallest form (nothing posted),
which is also the answer to a 413 from sending a value per individual.
|
|
A caller posting a mapping result back to /plot/rqtl -- or a compute
request carrying a value per individual plus covariates -- was refused
with `413 Failed to buffer the request body: length limit exceeded`. That
is axum's DefaultBodyLimit, 2 MiB, which nothing here had raised: a real
GEMMA run on the BXD WGS panel is ~1.9 MB, so the plot endpoints sat just
under the cap and anything bigger (a genome-sized run, or every animal's
covariate value) fell over it. nginx's 10m was never the binding limit.
* src/lib.rs (BODY_LIMIT_BYTES): 64 MiB, with the reasoning and the note
that a proxy has to allow as much or it refuses first.
* etc/nginx-vesuvio.pu.ac.ke.conf, docs/nginx.md: client_max_body_size
64m, so the two limits agree instead of the proxy cutting in at 10m.
* README.md ("Request body size"): both caps, and that a request does not
need to be large -- `{"case_attribute": "Sex"}` has the server read the
attribute and align it, where posting a value per animal is pure payload.
Verified: the 11.9 MB body that was refused now returns an SVG; 67 MB
still gets axum's 413.
|
|
Every database-backed endpoint 500'd with `CodecError { IO error: Broken
pipe }` while the process answered /version quite happily. The cause was
ours: `pool()` turns off the driver's `check_health` (a ping at checkout,
dead connections replaced) to save a round trip -- sound against a
database 130 ms away, worth nothing against loopback, and it costs the
pool its self-healing. One server-side close of an idle session then
becomes one failed request per dead socket: ten in a row, until a
restart.
* src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping;
`is_connection_loss` recognises the errors that mean the peer went away
(broken pipe, reset, went away, lost connection, closed);
`forget_pool` drops the cached pool so the next call rebuilds it;
`health` probes reachability, cached ten seconds. One pool static shared
by pool()/forget_pool (they had drifted into two).
* src/error.rs (ApiError::from_db): on a connection loss, forget the pool
-- the one place every database error passes through, so recovery does
not depend on each caller.
* src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a
service that answers while nothing behind it works is now visible.
* src/config.rs: the switch, with the trade written down.
* tests/db_recovery.rs: kills every pooled session from a second
connection, then asserts the next twelve requests all succeed -- the
scenario that produced the outage, now a regression test.
* tests/...: unit test for the classifier (its own IO error text and the
"this is a bad query, do not retry" cases).
* README.md ("Database connections"), docs/www/operator-checklist.md:
the knob, why it defaults on, and what /version reports.
|
|
R/qtl has always been able to fit covariates; the API now passes them
through. `covariates` takes one or more specs, each either a case
attribute of the group by name -- which the server reads, aligns onto the
panel and codes -- or explicit per-sample numbers:
{"case_attribute": "Sex"}
{"case_attribute": "Diet", "map": {"HFD": 0, "LFD": 1}}
{"name": "age_days", "kind": "numerical", "values": [...]}
The coding is the server's job because the phenotype file is numeric:
text with 2-20 levels is coded 0..k-1 in sorted order and declared
`categorical` (the wrapper dummy-expands more than two levels), numeric
text goes through as `numerical`, and one level, more than 20 levels, or
an explicit map that misses a value is a 422 asking for a map rather than
an invented ordering. An attribute the group does not define is a 422
that lists the ones it has. Missing values become `x`/NA and R/qtl drops
those individuals.
Everything that decides the model goes into the cache key -- covariate
names *and* kinds, since a categorical covariate's dummy expansion is not
the same model as a numeric one -- and the response says what happened:
Covariate Sex: categorical coded F=0, M=1, 3281 of 3281 samples have a
value, 3281 with both the trait and the covariate
Measured on BXD-Longevity AgeAtDeath (3281 animals, WGS panel, 21056
markers): LOD 23.100 without the covariate, 23.522 with Sex, same peak
(rs30847416, chr 1, 5.0 Mb), ~15s more for the covariate run.
* src/rqtl.rs (Covariate, CovariateKind, code_case_attribute,
write_covarstruct_file): the coding rules, the extra pheno columns, the
covarstruct file, `--addcovar --covarstruct`, and the model's cache key;
five unit tests.
* src/routes/rqtl.rs (RqtlCovariate): the request shape, resolution
through the group's case attributes, duplicate/limit/ambiguity checks
(8 covariates at most), and the per-covariate note.
* tests/rqtl.rs: a real covariate run on UM-HET3 (894 markers, so the
test stays a test) asserting the coding note and the `--addcovar` line,
plus the unknown-attribute 422.
* README.md, docs/www/ai-instruction.md (4a): how to pass them, and that
the note's coding and complete-case count are the things to read.
|
|
The trait-values endpoint stays about trait values; a covariate needs the
study's metadata for the same samples, aligned the same way. So:
GET /groups/{group}/traits/{trait_id}/case-attributes/{attribute}
returns one attribute's values for exactly that trait's samples, **in the
trait's own order**, which is what makes it usable without the caller
re-sorting anything:
/groups/BXD-Longevity/traits/10001/case-attributes/Sex
-> 3281 values in AgeAtDeath's order, F x 3101, M x 180
-> order identical to /traits/10001/values (asserted in the tests)
Alongside the values it reports `distinct` (each value with its count,
commonest first) and `num_values`. That is the check that decides whether
an attribute can serve as a covariate at all -- a constant one cannot --
and it saves a second request just to find out.
Attributes are per group, so an unknown one is a 404 whose `next` is the
group's attribute list: BXD has no Sex, BXD-Longevity does.
* src/db.rs (get_trait_case_attribute, get_group_trait_samples): the
group's attribute values aligned onto the trait's sample list, joining
CaseAttribute/CaseAttributeXRefNew on (InbredSetId, CaseAttributeId) as
the group endpoint does; empty text is reported as no value.
* src/routes/groups.rs: the handler, token-gated, with the response
shapes and the two `next` links (the trait's values, the group's
attributes).
* tests/case_attributes.rs: alignment against an independent /values
response, `distinct` accounting, ordering, and the 404s for BXD (no Sex)
and for a non-existent attribute.
* README.md, docs/www/ai-instruction.md (3. Case attributes): the
endpoint, and "ask per trait rather than sorting the group's attributes
yourself".
|
|
The §3 table had one line for /groups/{group}/case-attributes and nothing
else, so an assistant could find the endpoint and still not use it: the
response shape, the fact that the values are free text (`"Sex": "F"`,
`"Status": "Live"`), that `strain` is the group's mapping unit rather
than always a strain line, that which attributes exist is a property of
the study (BXD has no Sex; BXD-Longevity and eight other BXD-family
studies do), and that a published trait is the easier source when the
study has one.
Written around the worked case we settled on: AgeAtDeath in
BXD-Longevity, sex per animal available as a case attribute or as trait
10010 (1.0/0.0), and why that covariate matters there -- it varies
within the panel and has its own QTL (LRS 76 at rs13481466).
* docs/www/ai-instruction.md (3. What you can read): "Case attributes",
ahead of Search, matching the table's order.
|
|
The plot endpoints ran the mapping themselves, which meant the picture
could come from a different run than the one the caller was looking at,
and cost a scan to draw. They now take the mapping's own response -- the
JSON /compute/{rqtl,gemma} returned, posted back -- and hand it straight
to the plotter, which is what the scripts were written for
(`python3 plot-rqtl.py result.json -o out.svg --trait …`).
Validation replaces the run: `/plot/rqtl` needs `data.markers` with
positional 4+ column rows, `/plot/gemma` needs `data` rows *and*
`meta.columns` (its plotter reads the column names to find `p_lrt`), and
anything else is a 422 saying "post that response back". `/plot/rqtl`
also accepts just the `data` member; GEMMA cannot, and says why.
The mapping's `actions` suggestion now describes posting its own response
and carries no body -- duplicating a 21 000-row response into a
suggestion would double the envelope for nothing, and the caller already
holds it.
* src/routes/plot.rs: posted result, shape validation, `?trait`/`?title`
and `?bonferroni=false` passthrough to the scripts; no compute imports.
* src/routes/{rqtl,gemma}.rs: the plot suggestion loses its body.
* tests/plot.rs: canned results instead of real mappings -- the tests now
need only the plotter, no Rscript or gemma; covers both engines, the
bare-`data` form, and the 422 for a body that is a request.
* README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md:
"post the response back", and why GEMMA needs `meta.columns`.
|
|
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a
saved mapping response into an SVG Manhattan panel. They are now
endpoints, and the mapping that produces the response suggests them:
POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml
POST /plot/gemma same body as /compute/gemma -> image/svg+xml
Taking the mapping's own body rather than the mapping's output means one
call gets you the picture, and the plot cannot describe a run you did not
do: the handlers call the compute handlers and hand the resulting
envelope to the script, exactly as the scripts' docstrings describe
(they read `data.markers`, `data.permutation` and the notes' strain
counts). The mapping is re-run, but through the engines' own caches --
seconds on BXD.
The reply is an SVG, not the JSON envelope: an image is not JSON, and the
plotter's one-line peak summary rides back in `x-plot-summary` so a
caller that cannot see the picture still learns where the peak is.
Both compute responses gain the matching suggestion: `actions` carries
"the same request, plotted" (built from the request that ran, like the
permutation suggestion), so an assistant can map, look, and only then
decide whether the peak is worth the 100-permutation run.
* src/plot.rs: run the configured script over the envelope, read the
SVG back, clean up; a missing script is a 502 naming interpreter and
script, not a panic.
* src/routes/plot.rs: the two handlers (token-gated), delegating to
compute_rqtl/compute_gemma so validation and alignment stay in one
place; `x-plot-summary` sanitised into a header.
* src/routes/{rqtl,gemma}.rs: the plot suggestion, before the
permutation one.
* src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER.
* tests/plot.rs: the suggestion resolves to a complete SVG with a peak
summary, and an invalid mapping request is the mapping's own 422.
* tests/{rqtl,phewas}.rs: find suggestions by href rather than position.
* README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md
(new "Plotting": python3 must be on the server's PATH).
|
|
A scan gives LOD scores; the thresholds that make them readable come from
permuting the phenotype, which R/qtl supports (nperm) and this API
already ran -- but nothing told a caller to run it. So an unpermuted
R/qtl scan now ends with the same scan at nperm=100 in `actions` (a POST,
so not a `next` link), rebuilt from what actually ran: group + version +
the caller's own named samples, or the genotype file. The description
says what the thresholds are for and that it costs about 100 scans.
GEMMA is the honest half: it has no permutation test, and this endpoint
passes it nothing that would create one, so there is no nperm to suggest.
Its response says so in a note -- judge a peak against the smallest
p_lrt, not a genome-wide threshold -- and its `actions` suggests the
R/qtl permutation run over the same group and values instead. That is
real and runnable rather than a flag that would do nothing.
(gemma-wrapper does have `--permutate n`, which shuffles the phenotype
file and writes a permutation-stats file; wiring it up is a separate
job -- docs/genotypes.md and the README say it is not passed.)
* src/routes/rqtl.rs: the nperm=100 suggestion when nperm was 0, built
from the request that ran (original named samples, not the aligned
panel-length vector -- thousands of entries on a real panel).
* src/routes/gemma.rs: the note, and the cross-engine suggestion.
* tests/rqtl.rs: a path-based scan's suggestion reuses the same files.
* tests/phewas.rs: a group-scoped scan suggests both -- the PheWAS in
`next` (resolved) and the permutation run in `actions`.
* README.md, docs/www/ai-instruction.md (4a): what thresholds are, that
R/qtl offers them and GEMMA does not, and what to do about it.
|
|
A mapping's job is not finished when the rows come back: the next two
questions are "what else peaks here?" and "show me". So a group-scoped
mapping response now ends with both --
- `next`: `/groups/{group}/phewas?locus=<peak>&window=10000`, the peak
being the strongest LOD (R/qtl, column 4) or the smallest p_lrt (GEMMA,
last column). A GET that resolves 200, which is what `next` promises;
measured: R/qtl's peak rs48756159 -> 3 traits, GEMMA's
rsm10000005700 -> 12.
- a note saying which two columns to plot: chr against LOD (R/qtl) or
against -log10 of p_lrt (GEMMA). Rows are positional with meta.columns
naming them, so plotting needs no reshaping -- the API does not render
images, and saying exactly what to plot is more useful than pretending
to.
Path-based requests get neither: without a `group` there is no group to
PheWAS over, and the note alone would be noise.
* src/rqtl.rs, src/gemma.rs (peak_marker): strongest LOD / smallest
p_lrt, with unit tests for both shapes and the empty cases.
* src/routes/{rqtl,gemma}.rs: the `next` link and the plot note, when the
request named a group.
* docs/www/ai-instruction.md: 4b rewritten around what a reader does with
a PheWAS -- where the position comes from (a mapping's `next`, a
trait's `locus`, or chr+mb), the parameter table, what a hit contains,
and how to read it (the 460 LRS cap, `traits: []` versus `markers: []`,
peak-only and published-only limits, several traits on one marker),
with the chr 9 example; 4a points at the new forward.
|
|
PublishXRef records the marker each published trait's QTL peaks at
(Locus) with its LRS, which is a PheWAS waiting to happen: fix a
position, find the markers within a window of it, and ask which traits
peak at those markers.
GET /groups/{group}/phewas?locus=rs30524230&window=10000
GET /groups/{group}/phewas?chr=9&mb=29.940961&window=10000
The window is in base pairs, centred, default 10 kb either side -- the
"10K up and downstream" case. Positions come from the group's genotype
marker table (the endpoint added last round), so `genotype_version`
selects whose markers define the window and the answer reports them
alongside the traits: an empty `traits` list is then readable as
"nothing peaks that close" rather than a failure, and a ten-times wider
window is offered as `next`.
Measured on real data, ±10 kb around rs30524230 (chr 9, 29.94 Mb, a BXD
QTL peak at LRS 460): 4 markers, 10 traits, from LOD 99.9 down to 2.68 --
ANPEP_band, then microbiome/haematology/infectious-disease traits. That
cluster of 99.9s is the 460 LRS cap, not a spectrum, which the notes say.
* src/db.rs (get_phewas_traits): PublishXRef rows for a set of peak
markers, joining Phenotype/Publication through the existing
trait_info_sql, strongest first; marker names are re-validated before
they reach the IN list.
* src/routes/phewas.rs: the endpoint, token-gated; window/centre
validation (422 for none or negative), 404 for an unknown locus, notes
for the empty cases, and a unit test for the version/group helper.
* tests/phewas.rs: markers in the window, traits sorted by LRS with
lod = lrs/4.604, every hit peaking at a listed marker, zero-width window,
the two failure modes, and that the suggested wider window resolves.
* README.md, docs/www/ai-instruction.md (new 4b: how to read a PheWAS),
docs/www/operator-checklist.md: the endpoint and its reading.
|
|
* README.md (/version): the example carries dirty/source_digest/branch,
and the prose says why a stamp can lag a commit -- a binary built from a
dirty tree must not claim that commit -- and that build.rs re-stamps on
a commit because it watches the ref, not just .git/HEAD.
* docs/www/ai-instruction.md: comparing against a clone means comparing
the part before "-dirty", with source_digest for the exact build.
* tests/version.rs: `dirty` mirrors the suffix, branch is non-empty, and
source_digest is 16 hex digits or unknown.
|
|
The listing named versions but not files; a caller could pick one and
still not see what was in it. Three token-gated routes now describe a
version completely, short of the matrix:
GET /groups/{group}/genotypes versions, files, sizes, current
GET /groups/{group}/genotypes/{version} file name, header labels,
individuals, strains, chromosomes
GET /groups/{group}/genotypes/{version}/markers markers (name, chr, cM, Mb), paged
The detail reads the file's own `@`-labels (type, mat, pat, het, unk,
scale) and its panel in column order, and reports both the whole panel
and the strains a mapping uses after the parents/F1s rule -- so a caller
can check a request before sending it and interpret a result after.
Markers come back as positional rows with `meta.columns`, paged with
`?page`/`?limit` and filterable with `?chr`, following /compute/gemma's
shape rather than inventing one.
The genotype calls stay on the server: file_meta() reads the matrix to
skip it, and nothing returns a cell.
* src/genotypes.rs (Marker, FileMeta, file_meta; GenotypeFile::file):
header labels, individuals and marker positions; unit test.
* src/routes/genotypes.rs: the two new handlers, token-gated like the
listing, plus `file` in the listing and counts in the detail.
* tests/genotypes.rs: the listing carries file names but no paths.
* README.md, docs/genotypes.md, docs/www/ai-instruction.md: what the
three routes give, and that the matrix is not among them.
|
|
Until now /compute/{rqtl,gemma} took server paths, so the caller had to
know the deployment's layout, pick a version, and expand a trait's values
onto the panel in the file's order -- none of which the API told it. The
listing endpoint and the group-scoped request remove all three:
GET /groups/{group}/genotypes versions, newest first, no paths
POST /compute/rqtl {group, [genotype_version], samples}
POST /compute/gemma {group, [genotype_version], samples}
The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the
group's genotype/<Group>.json `genofile` list is newest-first and its
first entry is current; groups with no JSON (HET3-ITP) fall back to the
highest version number, and the listing's notes say which happened --
reads the panel from the .geno header the way GN3 does, and excludes
parents and F1s by default (GN2's `incparentsf1` off, GN3's
`get_sample_list(inc_par=False, inc_f1=False)`).
Excluding them has to subset the genotype matrix, not null-fill the
values: a null keeps an individual out of the fit but not out of GEMMA's
kinship or R/qtl's cross. So `prepare` writes a panel-subset file under
the engine's cache directory, keyed by panel, and reuses it.
Nothing in the listing or the mapping response carries a path: the
endpoint reports versions and sizes, and `notes` names the resolved
version, the panel size and how many values aligned. Explicit
`genotype_file`/`snps_file` remain as an override for files outside the
layout, and the two request shapes are mutually exclusive (422).
* src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1
exclusion, subset preparation, name alignment; 10 unit tests.
* src/routes/genotypes.rs: new endpoint, with a test that its list has no
path or file name in it.
* src/routes/{rqtl,gemma}.rs: group + genotype_version +
include_parents_f1; named `samples` aligned server-side; notes record
what resolved. gemma also accepts named samples where it used to insist
on positional trait_values.
* src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now
carry `group` and no placeholders; the dataset-scoped ones keep the
placeholder and say to add the group.
* src/routes/groups.rs: the group record points at /genotypes.
* src/config.rs: GENOTYPE_FILES (default the gn4-data directory).
* tests/genotypes.rs: new integration test (skips without the directory).
* README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the
versions, map by group" -- measured runs, the parent/F1 rule, the
subsetting requirement, and the restart list unchanged.
|
|
Paths travel in the request and no panel is cached in-process; both
engines key their caches on file contents (rqtl hashes the genotype file
into the output name, gemma-wrapper hashes into its kinship cache), so a
changed file is recomputed rather than silently reused.
The restart list is the useful half: new binary, GENENETWORK4_TEST_TOKEN
(decided when the router is built), and the environment fixed at first
use -- PATH/R_LIBS_SITE for R, SQL_URI's pool, the Xapian handle -- while
edits to scripts/rqtl_wrapper.R take effect immediately, since it runs
per request.
* docs/genotypes.md: "No restart is needed to use a new genotype file".
|
|
Asked whether the BXD mappings ran GN2/GN3's default (no parents, no
F1s): they did not. BXD.8's panel is 198 RI strains + 37 F1s and no
parent columns at all, and the runs passed all 235 with `null` for the
F1s -- phenotypes right, but GEMMA's kinship and R/qtl's cross spanning
235 individuals instead of 198.
Measured on trait 10001, both engines, both panels: identical answers
(GEMMA p 5.9e-07 at rsm10000005700 chr 8, R/qtl LOD 2.9320 at
rs48756159 chr 8), so nothing published from those runs is wrong. The
lesson is that null-filling excludes an individual from the fit but not
from the model; excluding it means subsetting the genotype file.
* docs/genotypes.md: "Parents and F1s are excluded by default" -- the
GN2 (`incparentsf1`) and GN3 (`get_sample_list(inc_par=False,
inc_f1=False)`) mechanisms, what each panel actually holds, the
both-ways measurement, and the rule for the alignment work.
* docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: the
same, in a sentence and a paragraph respectively.
|
|
docs/genotypes.md explained the convention; the endpoint docs still left
a caller guessing which file to pass and how big the panel is. So:
* README.md (/compute/gemma): GEMMA wants the BIMBAM pair
`genotype/bimbam/<file>_geno.txt` + `<file>_snps.txt`, not the `.geno`
file; `trait_values` is one entry per individual in that file's column
order; BXD's current panel stated (235 x 21056), with the 17.1 s
measurement against the 19 s the 7 320-marker fixtures take.
* README.md (/compute/rqtl): the file is `genotype/<file>.geno`
(BXD.8: 235 x 21057; HET3-ITP.12: 6447 x 894), `MappingMethodId`
decides the engine, and the two measured runs with the UM-HET3 peak.
* README.md (/compute/gemma/ravanan): same input convention as gemma.
* docs/genotypes.md: how each measurement was made, step by step, so
they can be repeated; and that step 3 (expanding values onto the panel)
is the API's job in the next iteration.
* docs/www/ai-instruction.md (4a): a table of the BXD files per engine
with panel sizes, the UM-HET3 file, "ask the operator for any other
panel" with where GN2 records which one is current, and the measured
times. Values must be one per individual, in file order -- position, not
name -- and the API does not re-index yet, so a mapping is prepared for
the caller rather than guessed.
* docs/www/operator-checklist.md: MappingMethodId per group, and the
three measurements.
|
|
The compute endpoints take server-side paths, so callers need to know
the deployment's genotype-file layout and which version is current --
neither of which is discoverable from this repository, and both of which
are spread across GN2 and GN3 (GN2's per-group genotype/<Group>.json
whose `genofile` list is newest-first and is the actual authority -- the
GenoFile table still lists only the classic files; GEMMA fed from
genotype/bimbam/<file>_geno.txt, R/qtl from genotype/<file>.geno; GN3
reading the sample order out of the .geno header; InbredSet.
MappingMethodId deciding which engine a group offers).
Measured on this deployment, with the newest genotypes and values
re-indexed onto each panel's sample order:
BXD BXD.8.geno (WGS Mar2022) GEMMA 17.1s 21055 rows
BXD BXD.8.geno R/qtl 7.5s 21056 markers
HET3-ITP.12 (Apr2023, 6447 individuals) R/qtl 83.1s 893 markers,
lifespan peak LOD 7.95 at 1_24042124 (chr1, 24.04 Mb)
That last run is the shape of what Rob Williams wants: UM-HET3 is a
4-way cross (MappingMethodId=3, R/qtl), and its genotype files are
versioned per cohort with no JSON recording which is current, and no
per-cohort trait split -- 6438 of lifespan's 23879 animal values fall in
the .12 panel, the rest are silently dropped. Recorded as the known
messiness rather than hidden.
* docs/genotypes.md: new. Layout, how GN2/GN3 choose, the alignment
rule, per-group engines, the measurements, what is still messy.
* README.md: pointer.
* docs/www/operator-checklist.md: replace "only the test fixtures exist"
with the real panels, and name the missing alignment step.
|