| Age | Commit message (Collapse) | Author |
|
tests/envelope.rs follows every href in every reply, and running it end to
end found six that were not addresses:
- /species, /species/{name}/datasets and /groups/{group}/datasets offered
/species/{species} and /datasets/{dataset} -- templates, which 404 when
followed. Each now points at a real row from the listing in hand, and
the description says any other name can go in its place.
- /datasets/{name} offered /datasets/{dataset}/probesets/{probeset}: the
listing it already links is the callable one, so the shape moved into
that link's description.
- /traits/{dataset}/{trait}/values offered a GET
/traits/{ds}/{trait}/correlation, which does not exist: correlation is
POST /compute/correlation and that reply already carries it in
"actions". Removed, with the reason written where it was.
- the same reply built its /groups/... links from the *dataset* name.
BXDPublish's traits live in group BXD, and /groups/BXDPublish/phewas is
a 404 (no genotype files for it). db::get_group_name resolves the group
from InbredSet.Id and the links use it; when a dataset has no group the
group links are simply absent.
- /groups/{g}/phewas with no parameters is a 422 ("give either locus, or
chr and mb"), so that link is now a shape with placeholders.
Placeholders stay where the caller must supply a value
(.../markers/{marker}/classes?trait={trait_id}), but the crawl now skips
them only when the description says what to fill in -- otherwise a client
reading the next list cannot tell a call from a template.
Two stale expectations alongside: /version carries the root links now (it
is the natural first call), and /search's results put its own follow-ups
first and the root links after them.
Tests are honest about their environment too: links skips when no search
index is configured (an empty XAPIAN_DB_PATH answers 502), and the
envelope crawl exempts /openapi.json, the API document, which is
deliberately not enveloped.
|
|
The gotcha notes in SKILL.md tell an agent to count the animals per
genotype class before believing a peak; a note in a document is only
found by whoever reads it, so the replies now say it too, at the peak
they just computed.
* suggest::class_hint builds
/groups/{g}/genotypes/{v}/markers/{peak}/classes?trait={trait_id} --
a real GET with one placeholder, because a scan knows the trait's
values, not its PublishXRef.Id.
* /compute/{rqtl,gemma,qtl2,fitqtl} add it to `next` after the PheWAS
link, naming the peak each reply found and the version the request
named (a {version} placeholder when it named none).
* The genotype subtree carries the template too: a version's detail,
and a concrete marker's own links.
* /compute/qtl2 gains the sentence that explains the divergence: on a
riself (riset) panel heterozygous calls are dropped before the scan
(R/qtl's convert2riself) and the HMM re-imputes them, while GEMMA
keeps them at dosage 0.5, so a mostly-H marker cannot peak there and
two engines can name different markers of one region without either
being wrong.
tests/links.rs: a scan's reply offers the class counts at its own peak --
the link names the version and the peak marker the reply reports, and
leaves {trait_id} for the caller.
|
|
A caller that knows only /version should be able to reach anything by
following links. Until now `next` was thin and uneven: the genotype
subtree offered none at all, the group record offered one, /version
offered nothing, and two links it did offer could not be followed.
* suggest:: now builds the graph: root_links (groups, species,
mapping-methods, openapi), genotype_version_links (frequencies,
marker names, a marker's location and tally, the group, the scanning
actions), unlisted_genotype_links, marker_links (location, tally,
marker list, panel tally, PheWAS at that marker), group_links and
trait_links. Envelope::with_links / with_links_except add them,
deduplicating and never suggesting the response the caller is
already reading.
* Wired through the genotype surface (listing, version detail, markers,
location), both frequencies routes, the compute endpoints, groups,
traits, both value routes, species, datasets, publications,
accessions, search, mapping methods, phewas, correlation and
/version.
* Fixed what was not followable, because `next` promises a GET that
returns 200 and a client is the one that finds out otherwise:
- /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are
gone; comput endpoints now link ?locus=<their strongest marker>
(suggest::peak_marker_of, computed before the rows move).
- /datasets and /publications are not routes; root_links offers the
listings that exist (/groups, /species, /mapping-methods) and
/openapi.json.
- /search?q= with an empty query is a 400, so it is not a link.
- the correlation endpoint's "/compute/correlation" was a POST in
`next`; it is in `actions` now.
- marker_links pointed at /markers/{marker}, which is not a route;
it points at /markers/{marker}/location.
tests/links.rs walks the graph two hops from a dozen seeds and fails on
any link that does not answer 200, plus a check that a tally leads to the
marker list and to a marker's location, and a location to that marker's
tally and its PheWAS.
|
|
GeneNetwork2's trait page offers a set of normalisations, and its mapping
page warns about outliers without ever fixing them. This adds both, under
one spelling that `/compute/rqtl`, `/compute/gemma` and
`/groups/{g}/traits/{id}/values` all share:
normalize none | log2 | log10 | sqrt | negate | zscore |
robust_zscore | rank (GN2's names `invert` and
`qnorm` accepted)
winsor_quantile clip each tail at a fraction, e.g. 0.05
winsor_fences clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule
They run in a fixed, reported order -- transform, then winsorise, then
standardise -- so clipping acts on the scale being analysed and the result
lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The
note gives the steps *in the order they ran* (`winsorized at Tukey fences,
then zscore`), the count clipped in each tail, and how many values sit
outside Tukey fences -- the warning GN2 gives before mapping and this API
gave nowhere.
**Checked against genenetwork.org itself.** GN2's public trait page embeds
its own reference values (`js_data.qnorm_values` / `zscore_values`, the
`data-qnorm`/`data-zscore` attributes), so there is an oracle for the two
transforms whose definition matters. On BXD trait 10010 (28 samples):
`rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's
three-decimal rounding. Getting there found a real disagreement: GN2 uses
`scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use
the **population** sd; ours used the sample sd and came out
`sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2.
Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based
inverse-normal (Blom) transform, so it is called `rank` here; and its
`sqrt` silently adds 1 when a value is in [0,1], which shifts every effect
size -- a negative value is refused instead.
* src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom
scores over an Acklam inverse-normal, `Params` for query/body use,
`to_json` for suggestions, and eleven unit tests -- the quantile table,
ties sharing an average rank, missing values never imputed, GN2's own
`find_outliers` example, and the note's wording.
* src/routes/{rqtl,gemma}.rs: the three fields, applied **after**
alignment (over the samples that enter the fit) and before anything reads
the values, with the note; the R/qtl permutation suggestion and GEMMA's
R/qtl suggestion carry the pipeline.
* src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values
endpoint, and the mapping bodies in `actions` carry the same fields -- a
caller who fetches z-scores must map those numbers, not the raw ones.
* tests/groups.rs: values are mean 0 / population sd 1, the note says so,
every `/compute/*` suggestion in `actions` carries it, and an impossible
pipeline is a 422 `invalid_normalization`.
* README.md, docs/www/SKILL.md: what each name means, the order, the
population-sd parity, and that effects come out on the transformed scale.
298 tests pass; the same four as before do not.
|
|
Until now /compute/{rqtl,gemma} took server paths, so the caller had to
know the deployment's layout, pick a version, and expand a trait's values
onto the panel in the file's order -- none of which the API told it. The
listing endpoint and the group-scoped request remove all three:
GET /groups/{group}/genotypes versions, newest first, no paths
POST /compute/rqtl {group, [genotype_version], samples}
POST /compute/gemma {group, [genotype_version], samples}
The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the
group's genotype/<Group>.json `genofile` list is newest-first and its
first entry is current; groups with no JSON (HET3-ITP) fall back to the
highest version number, and the listing's notes say which happened --
reads the panel from the .geno header the way GN3 does, and excludes
parents and F1s by default (GN2's `incparentsf1` off, GN3's
`get_sample_list(inc_par=False, inc_f1=False)`).
Excluding them has to subset the genotype matrix, not null-fill the
values: a null keeps an individual out of the fit but not out of GEMMA's
kinship or R/qtl's cross. So `prepare` writes a panel-subset file under
the engine's cache directory, keyed by panel, and reuses it.
Nothing in the listing or the mapping response carries a path: the
endpoint reports versions and sizes, and `notes` names the resolved
version, the panel size and how many values aligned. Explicit
`genotype_file`/`snps_file` remain as an override for files outside the
layout, and the two request shapes are mutually exclusive (422).
* src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1
exclusion, subset preparation, name alignment; 10 unit tests.
* src/routes/genotypes.rs: new endpoint, with a test that its list has no
path or file name in it.
* src/routes/{rqtl,gemma}.rs: group + genotype_version +
include_parents_f1; named `samples` aligned server-side; notes record
what resolved. gemma also accepts named samples where it used to insist
on positional trait_values.
* src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now
carry `group` and no placeholders; the dataset-scoped ones keep the
placeholder and say to add the group.
* src/routes/groups.rs: the group record points at /genotypes.
* src/config.rs: GENOTYPE_FILES (default the gn4-data directory).
* tests/genotypes.rs: new integration test (skips without the directory).
* README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the
versions, map by group" -- measured runs, the parent/F1 rule, the
subsetting requirement, and the restart list unchanged.
|
|
Fetching one trait's value per strain is usually the step before mapping
that trait, and the API can do the mapping. Those calls are POSTs with a
body, so they cannot go in `next` -- a `next` link is a GET that returns
200, which is what makes `next` safe to follow blindly and what
tests/envelope.rs relies on. Hence a fifth envelope member, `actions`,
holding method + href + description + body, and omitted entirely when
there is nothing to suggest (most responses keep the four-member shape).
The three per-trait values endpoints now carry a `POST /compute/rqtl`
and a `POST /compute/gemma` suggestion with the values already in the
body, in the response's own sample order. Genotype/SNP paths stay
bracketed placeholders: they must exist on the server, which the server
cannot know on the caller's behalf.
The instruction page says, in the same breath, never to run R/qtl or
GEMMA locally -- a local scan would use genotypes that are not the
panel's and would disagree with GeneNetwork's own results -- and that
results are cached, so repeat calls are cheap (measured: ~2 s cached
against ~20 s cold on the BXD fixtures).
Also fixes a flake the new test exposed rather than caused:
config::tests read and write process-global env vars and Config::load()
reads all of them, so the RAVANAN_* tests could fail
search_engine_keys_follow_env_over_default_layering on an unrelated
assertion. One module-level lock serialises them.
* src/response.rs (SuggestedCall, Envelope::actions/with_action/
with_actions): the new member, omitted when empty; unit tests for both.
* src/suggest.rs: new. Builds the two suggestions for either value type
(db::TraitValue, db::ProbeSetValue), with its own unit tests.
* src/routes/{traits,groups,datasets}.rs: the three values handlers.
* tests/compute_suggestions.rs: new. Suggestions present with values in
order, on all three routes; absent elsewhere; `next` still GET-only and
still returns 200.
* src/config.rs (tests): env_guard, taken by the 11 config tests.
* README.md (Success responses): the `actions` member, with an example.
* docs/www/ai-instruction.md (4a): never compute locally, why, caching,
and how to use the suggested bodies.
* docs/www/operator-checklist.md: the caching numbers.
|