| Age | Commit message (Collapse) | Author |
|
Until now /compute/{rqtl,gemma} took server paths, so the caller had to
know the deployment's layout, pick a version, and expand a trait's values
onto the panel in the file's order -- none of which the API told it. The
listing endpoint and the group-scoped request remove all three:
GET /groups/{group}/genotypes versions, newest first, no paths
POST /compute/rqtl {group, [genotype_version], samples}
POST /compute/gemma {group, [genotype_version], samples}
The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the
group's genotype/<Group>.json `genofile` list is newest-first and its
first entry is current; groups with no JSON (HET3-ITP) fall back to the
highest version number, and the listing's notes say which happened --
reads the panel from the .geno header the way GN3 does, and excludes
parents and F1s by default (GN2's `incparentsf1` off, GN3's
`get_sample_list(inc_par=False, inc_f1=False)`).
Excluding them has to subset the genotype matrix, not null-fill the
values: a null keeps an individual out of the fit but not out of GEMMA's
kinship or R/qtl's cross. So `prepare` writes a panel-subset file under
the engine's cache directory, keyed by panel, and reuses it.
Nothing in the listing or the mapping response carries a path: the
endpoint reports versions and sizes, and `notes` names the resolved
version, the panel size and how many values aligned. Explicit
`genotype_file`/`snps_file` remain as an override for files outside the
layout, and the two request shapes are mutually exclusive (422).
* src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1
exclusion, subset preparation, name alignment; 10 unit tests.
* src/routes/genotypes.rs: new endpoint, with a test that its list has no
path or file name in it.
* src/routes/{rqtl,gemma}.rs: group + genotype_version +
include_parents_f1; named `samples` aligned server-side; notes record
what resolved. gemma also accepts named samples where it used to insist
on positional trait_values.
* src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now
carry `group` and no placeholders; the dataset-scoped ones keep the
placeholder and say to add the group.
* src/routes/groups.rs: the group record points at /genotypes.
* src/config.rs: GENOTYPE_FILES (default the gn4-data directory).
* tests/genotypes.rs: new integration test (skips without the directory).
* README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the
versions, map by group" -- measured runs, the parent/F1 rule, the
subsetting requirement, and the restart list unchanged.
|
|
Fetching one trait's value per strain is usually the step before mapping
that trait, and the API can do the mapping. Those calls are POSTs with a
body, so they cannot go in `next` -- a `next` link is a GET that returns
200, which is what makes `next` safe to follow blindly and what
tests/envelope.rs relies on. Hence a fifth envelope member, `actions`,
holding method + href + description + body, and omitted entirely when
there is nothing to suggest (most responses keep the four-member shape).
The three per-trait values endpoints now carry a `POST /compute/rqtl`
and a `POST /compute/gemma` suggestion with the values already in the
body, in the response's own sample order. Genotype/SNP paths stay
bracketed placeholders: they must exist on the server, which the server
cannot know on the caller's behalf.
The instruction page says, in the same breath, never to run R/qtl or
GEMMA locally -- a local scan would use genotypes that are not the
panel's and would disagree with GeneNetwork's own results -- and that
results are cached, so repeat calls are cheap (measured: ~2 s cached
against ~20 s cold on the BXD fixtures).
Also fixes a flake the new test exposed rather than caused:
config::tests read and write process-global env vars and Config::load()
reads all of them, so the RAVANAN_* tests could fail
search_engine_keys_follow_env_over_default_layering on an unrelated
assertion. One module-level lock serialises them.
* src/response.rs (SuggestedCall, Envelope::actions/with_action/
with_actions): the new member, omitted when empty; unit tests for both.
* src/suggest.rs: new. Builds the two suggestions for either value type
(db::TraitValue, db::ProbeSetValue), with its own unit tests.
* src/routes/{traits,groups,datasets}.rs: the three values handlers.
* tests/compute_suggestions.rs: new. Suggestions present with values in
order, on all three routes; absent elsewhere; `next` still GET-only and
still returns 200.
* src/config.rs (tests): env_guard, taken by the 11 config tests.
* README.md (Success responses): the `actions` member, with an example.
* docs/www/ai-instruction.md (4a): never compute locally, why, caching,
and how to use the suggested bodies.
* docs/www/operator-checklist.md: the caching numbers.
|
|
mark_failed_appends_failure_marker_with_the_error and
create_run_log_rejects_a_reused_id both worked in
/tmp/gn4-streaming-test-<pid>, and the second removes the directory when
it is done, so under cargo's parallel test threads it could delete the
first one's log mid-assertion -- an intermittent failure that has
nothing to do with either test's subject. Give each its own directory.
|
|
The index stores every boolean term in lowercase and holds no
mixed-case variants at all: XDSbxdpublish (15 060 docs), Awilliams
(10 958), XSmouse, XCx, XYbdnf, XImrna all exist, while XDSBXDPublish,
AWilliams, XSMouse, XCX, XYBDNF and XIMRNA report "not in database" on
both the dev index and a full production one. Matching the value
exactly therefore rejected the way a caller naturally writes it --
dataset:BXDPublish returned 0 hits against an index holding 15 060
BXDPublish traits, author:Williams 0 against 10 958 williams authors.
Lowercase the value for every boolean prefix, generalising the
group-only rule gn3 already has (search.py:204-209). No graded parity
shape changes: they all use lowercase values already.
Pinned per prefix in boolean_prefixes_parse_to_exact_index_terms, so a
registration that regresses to an exact add_boolean_prefix fails there
instead of silently returning zero hits.
|
|
The cache dir was keyed by chromosome only and shared across requests,
so a request with different inputs (another trait, or another snps or
genotype file) left a second *-gemma-cXX.tar.xz in the same chr-N dir.
gemma-wrapper names those archives by a content hash, so the workflow's
single-File glob output binding then matched more than one path and the
step failed -- silently, since ravanan's user-error only logs when
%log-level is set, which never happens inside a generated job script.
Key the cache root by a hash of the input set (genotype path and size,
snps contents, trait values, lmm) as well as by chromosome, so repeated
identical requests still hit gemma-wrapper's cache while different
input sets cannot see each other's archives. chromosomes_in reads the
snps file, so it now takes the already-read contents (chromosomes_from)
to avoid reading it twice.
cwl/README.md: describe the two-part keying, and note that a node needs
a writable HOME -- ravanan forwards the submitting process's HOME into
the job (as sbatch would) and gemma-wrapper writes its lock file there.
|
|
POST /compute/gemma/ravanan used to run every CWL step locally, because
compute_ravanan never passed ravanan a batch system. Add a
RavananBatch setting:
- RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing
closed on anything else, exactly like GN4_DB_MODE;
- slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint,
--slurm-jwt (a *file*, which is what ravanan reads) and, when set,
--slurm-partition/--slurm-nice; a slurm selection without an endpoint
or a JWT file refuses to start rather than 502-ing every request;
- --scratch defaults to ravanan_cache_dir for slurm, so the store and
the job scratch space live on the same shared filesystem (with slurm
that directory has to be visible to the compute nodes);
- the response's ravanan note records which backend ran.
The switch building is RavananBatch::ravanan_args so it is unit
testable; compute_ravanan takes the backend and the scratch directory,
and the route passes them from config. docs/ravanan-slurm.md documents
the keys, the shared-directory requirement, and the slurm-side auth
wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in
slurmrestd's environment) that makes REST-submitted jobs actually run.
|
|
# Conflicts:
# src/gemma.rs
# src/rqtl.rs
# src/streaming.rs
|
|
gemma-loco.cwl was a generated artifact (ccwl compile's output), but
was being committed like hand-written source. Remove it, and have
compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL
file under its own cache dir on every call instead -- cheap (pure
macro expansion, no Guix build involved), so there's no real cost to
regenerating it fresh each time, and no risk of a stale compiled file
drifting from its ccwl source.
Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH,
alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now
names the .scm source, not a .cwl file. Verified end to end by running
the exact compile-then-run sequence compute_ravanan now performs
against the real ccwl/ravanan/gemma-wrapper binaries and the BXD
fixtures: same correctness and timings as before.
|
|
Per Arun's request, express the --loco kinship/GWA workflow as a
ccwl (https://ccwl.systemreboot.net/) source, cwl/gemma/gemma-loco.scm,
compiled with `ccwl compile --output=gemma-loco.cwl gemma-loco.scm`.
ccwl inlines the two command definitions (kinship, gwa) directly into
the Workflow CWL, so tools/kinship.cwl and tools/gwa.cwl are no longer
needed as separate files.
The two scatter steps (kinship over [chr, cache_dir], gwa over [chr,
cache_dir, kinship_json], both dotproduct) and the gwa_json/archive
outputs are unchanged in substance -- verified by compiling and
running the generated gemma-loco.cwl through ravanan against the BXD
fixtures: same 20-chromosome scatter, same cold (~17.7s) and warm
(~2.4s, 0 jobs re-run) timings, and the same GWA values (rs31443144:
chr=1, ps=3010274, p_lrt=0.2178407) as the hand-written CWL it
replaces.
|
|
--cache-dir "." pointed every scattered job at its own hermetic CWL
working directory (fresh under ravanan's --store every run), so
gemma-wrapper's cache was never actually reused across requests.
Point it instead at <ravanan_cache_dir>/gemma-wrapper-cache/chr-<N>,
one persistent directory per chromosome -- not a single shared
directory, because gemma-wrapper's cache-hit hash does not vary with
--chromosomes, so sharing one directory would let one chromosome's
job silently reuse another's cached result.
Also drops gwa.cwl's InitialWorkDirRequirement staging hack: since
kinship and gwa for the same chromosome now point at the same
persistent cache_dir, gemma-wrapper finds the kinship step's archive
there directly, no staging needed.
Confirmed via a real integration run (BXD fixtures) that gemma-wrapper
itself unconditionally re-executes GEMMA regardless of whether a
matching cached archive already exists -- --cache-dir is only used to
hand results between gemma-wrapper's own pipeline stages within one
request, not to skip recomputation across requests. So this fixes a
real latent bug (archive-glob collisions once a chromosome's cache
directory holds more than one request's output) but does not, on its
own, speed up repeated identical requests; verified the direct
/compute/gemma path shows the same lack of cross-request speedup.
|
|
Replace gemma-loco.cwl's single kinship/gwa pair (each one gemma-wrapper
invocation covering all chromosomes, relying on gemma-wrapper's own
internal GNU-parallel fan-out) with one gemma-wrapper invocation PER
CHROMOSOME, restricted via its existing --chromosomes/--no-parallel
flags and scattered by ravanan itself (ScatterFeatureRequirement,
scatterMethod: dotproduct). gemma-wrapper still does the correct
per-chromosome LOCO math (exclude this chromosome from kinship, test
only its markers) against the full genotype/SNP files -- only which
chromosome(s) it loops over changes. compute_ravanan now computes the
chromosome list itself (chromosomes_in, mirroring gemma-wrapper's own
get_chromosomes) and passes it as the workflow's scatter input, and
collates the resulting per-chromosome gwa_json/archive array outputs.
Measured 16.7s wall-clock end-to-end against the BXD fixtures (20
chromosomes), vs. ~19s for /compute/gemma's direct path and ~21s for
the previous sequential ravanan workflow -- confirmed byte-for-byte
identical output (rs31443144's exact values, 7320 total rows).
Also fixes three real, previously-unexercised bugs in compute_ravanan
found getting this to run at all (POST /compute/gemma/ravanan had no
automated test until this commit):
- missing --guix-manifest (ravanan requires it; added
gemma_ravanan_manifest config key and cwl/gemma/manifest.scm)
- missing --store files/logs/data subdirectories (ravanan does not
create them itself)
- --output-directory dropped entirely: every chromosome's gwa archive
shares gemma-wrapper's own content-hash basename (the hash doesn't
factor in --chromosomes), so ravanan's optional flat-symlink step
collides across them; unneeded anyway since results are read
straight from ravanan's --store via its stdout JSON record.
Also found and worked around a ravanan scheduler bug: a step scattered
over exactly one input needs an explicit scatterMethod even though CWL
only requires one for multiple scattered inputs -- ravanan has no
fallback case for a missing one and silently misbehaves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
So a caller of /compute/gemma(/ravanan) or /compute/rqtl doesn't need
to count data's rows or columns.len() itself. with_columns now takes
the row count too and derives num_cols from the column list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Same space-saving shape as the gemma commit: /compute/rqtl's markers
are now positional arrays (name, chr, pos, lod, plus any 4-way-cross
extra effect columns trailing) instead of one keyed object per row,
with the fixed column names in meta.columns.
Also fixes tests/gemma.rs, which still asserted the old keyed shape
from the gemma change -- missed in that commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Cuts repeated key names out of large /compute/gemma(/ravanan) responses:
data is now a list of positional row arrays instead of one keyed object
per row, with the column names moved to a new meta.columns field shared
with any other endpoint that wants the same shape.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
|
|
- rqtl: per-call unique output filename. The content hash alone was
deterministic across processes, so two concurrent identical
requests shared one output file; R truncates on open, so one
request could read the other's half-written results with no error.
The per-call unique (pid + nanos, already computed for the pheno
file) now names the output file too.
- streaming/gemma/rqtl: append a failure marker to the run log on
every error path, mirroring the completion marker. A status
poller can now tell a failed run from a still-running one.
- rqtl: --scale is now honored. The wrapper selects the geno file's
cM/Mb map column from it, and the route rejects values other
than mb/cm with 422 instead of silently ignoring them.
Both new behaviors are unit-tested; the live Rscript tests pass.
|
|
Pure formatting sweep (import ordering, line wrapping); the tree
had accumulated rustfmt drift. Separated from any semantic change
so it stays reviewable and revertable on its own.
|
|
|
|
* src/config.rs (DbMode, Config.db_mode, parse_db_mode): add
GN4_DB_MODE=dev|production in the layered config; dev runs with no
configured DB; fail-closed parse (a typo refuses).
* src/db.rs (DbError, pool, probe), src/error.rs (ApiError::from_db):
mint a typed 503 db_unavailable at the db::pool() choke point,
distinguishing Unavailable from Query.
* src/main.rs: loud no-DB startup line; production refuses to start
(exit non-zero) before axum::serve.
* src/routes/{accessions,datasets,groups,mapping_methods,publications,species,traits}.rs:
route every DB error through ApiError::from_db.
* src/bin/gn-indexer.rs: refuse to build an index when the DB is unreachable.
* tests/no_db.rs: pin no-DB degradation (search 200 partial, DB routes 503).
His ask (DM 2026-10-12): "Make it an option to switch it off for
development machines? I mean mariadb." The no-DB hardening is ours
(review-driven). Default GN4_DB_MODE left unflipped (=dev); the flip needs
owner sign-off.
|
|
* src/indexer/delta.rs (scan): enumerate dataset labels under COLLATE
utf8mb4_bin so a case-only rename's old and new labels both enter scope;
route to FULL when a case-variant label pair is in the affected scope.
* src/indexer/run.rs (run_delta): seed the duplicate-check map with manifest
labels not in scope; route to FULL when a manifest carries an unknown
versioning flavour.
* scripts/gn-indexer-lag.sh: honour the table-prefix seam only with the
explicit test opt-in; treat a literal NULL MAX as no change; guard an
empty lag value.
* scripts/gn-indexer-run.sh: --index=PATH-aware argument scan.
|
|
* src/indexer/delta.rs: change detection over MariaDB system versioning
(ROW_START/ROW_END): coverage marker, versioning probe, purge check,
per-table candidate queries and identity-term fan-out.
* src/indexer/run.rs (run_delta): reuse the streaming crawl and shard diff,
scoped to the affected datasets.
* src/indexer/sidecar.rs: manifest v3 with versioning_marker/versioning_flavour.
* src/bin/gn-indexer.rs: add the delta mode.
* scripts/gn-indexer-run.sh, scripts/gn-indexer-lag.sh: flock wrapper and lag SLA.
|
|
GN4_INDEXER_HALT_AFTER_COMMITS parks the process and
GN4_INDEXER_COMMIT_CHUNK reshapes the commit cadence; either set by
accident in a production environment could hang or change a run. Make
both inert unless GN4_INDEXER_TEST_HOOKS=1.
|
|
shard_filename truncated the label's SHA-256 to 32 bits, so two labels
sharing a sanitized stem and a 32-bit prefix could merge into one shard
(last writer wins). Use 16 hex chars and assert length and distinctness.
|
|
Duplicate-identity detection kept only the current dataset's set, so two
labels differing only by case lowercased to the same <dataset> and
collided across shards unseen; the manifest doc_count would then
double-count and every later run would re-FULL forever. Keep an
O(#datasets) lowercased-label map and flag a second distinct raw label
that lowers to a seen key.
|
|
The crawl ordered by dataset with the Name column's utf8mb4_general_ci
collation while merge_streams compares labels by Rust byte order, so a
label could arrive in two non-contiguous runs and the second begin_dataset
would drop the first run's rows. Pin COLLATE utf8mb4_bin on both ORDER BY
clauses so the SQL order matches the merge.
|
|
A monolithic sidecar rewrote and re-read the whole identity map each run;
sharding it by dataset under <index>.sidecar.json plus <index>.sidecar.d/
lets a run load and rewrite only the shards it touches and keeps resident
memory at O(largest dataset).
* src/indexer/run.rs: crawl the SQL sources one dataset at a time through a
sink and commit every 5000 documents instead of buffering the whole crawl;
a bounded-channel variant was tried and reverted (several times slower).
* src/indexer/rows.rs: order and group the crawl streams by dataset.
* src/bin/gn-indexer.rs: follow the sharded sink wiring.
|
|
Building and writing each document as the crawl yields it is leaner than
collecting the whole set in a Vec, so live documents stay O(1) and peak
memory stops scaling with the corpus.
* src/indexer/livemem.rs: the live-document counter that proves it.
* src/indexer/build.rs, src/indexer/mod.rs, src/bin/gn-indexer.rs: adapt
to the streamed build and write signatures.
* Cargo.lock: repin xapian-rs 97fd343 -> 70c5eac (the write-API set).
|
|
Adds a gnauth-macros proc-macro crate and a private-field Authorized
capability type in src/auth.rs, so #[gnauth(secured)] is a real,
compile-time-checked guarantee that a handler performs the gn-auth
resource-authorisation round trip before sharing trait/probeset
values, rather than just a comment. #[gnauth(token_only)] marks
handlers that require a token but skip the round trip, and
#[gnauth(disabled)] marks the two handlers that need no token at all.
Also fixes groups.rs: group_trait_values, group_trait_info and
group_traits previously performed no gn-auth round trip at all for
published-trait data; they now resolve the group's PublishFreeze row
and call the same auth::check used elsewhere, denying by default when
no such row exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Same request/response shape as /compute/gemma: synchronous (no job
queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming.
Extracted that streaming infrastructure (run_streaming, create_run_log/
RunLogError, status/StatusError, RunStatus/RunStatusParams,
is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new
src/streaming.rs so both compute endpoints share it instead of
duplicating it; src/routes/gemma.rs updated to the shared names,
confirmed its own 8 tests still pass unchanged.
scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim
and run via Rscript -- same reasoning as reusing gemma-wrapper rather
than reimplementing GEMMA. Scope: a single scan (scanone) plus, if
nperm > 0, its permutation thresholds (67th/95th percentile, via a
Rust port of numpy.percentile's default linear interpolation) -- the
common case. Not implemented: covariates, interval mapping,
permutation strata, pair-scan (scantwo) -- scantwo in particular needs
cross-referencing the genotype file for nearest-marker lookups and a
second (MAP_) output file, a materially bigger port than this covers.
One fix relative to GeneNetwork3, found by reading the code (no test
covers the real caller's keys): generate_rqtl_cmd's content hash only
ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller
uses keys "geno"/"pheno" -- they never match, so the genotype/
phenotype file contents never actually enter the cache-key hash in
production, only the other options. Two different phenotypes mapped
with the same other options would get the same cached output filename
and silently reuse each other's result. Fixed here: both files'
contents are always hashed (std DefaultHasher -- a cache key, not a
security boundary, so no need for GeneNetwork3's md5).
guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as
propagated-inputs (same category as gemma-wrapper, and same reasoning:
invoked as a subprocess, not linked). All stock Guix -- confirmed the
existing -L (needed only for gemma-wrapper, from guix-bioinformatics)
is not needed for these; the dev shell has Rscript/R-qtl on PATH from
guix.scm's own propagated-inputs alone.
Test fixture: a small, hand-written, valid BXD riset genotype/
phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's
own tests/unit/db/.../genotype_sample1.geno -- that fixture has a
genotype code outside its own header's declared set and duplicate
marker names across chromosomes, harmless for the unrelated parsing
unit it's written for there, but both would break a real R/qtl run.
Verified end to end, not just unit tested: tests/rqtl.rs runs the real
Rscript/R-qtl against that fixture (a real scanone scan, real
permutation thresholds with suggestive <= significant, the run-id-
collision 409 pre-seeded with no Rscript needed, and the completion
marker landing in a real log after the run finishes). cargo test
passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
|
|
peak -> peek: the status endpoint's byte-offset query parameter and
everywhere it's documented (code comments, the GemmaStatusParams
field, tests, README). GeneNetwork3's own equivalent
(gn3/api/streaming.py) is spelled "peak"; noted where this now
deliberately differs, not touching the unrelated "peak marker" wording
elsewhere in the README.
Run id uniqueness: the log file is now created with
OpenOptions::create_new(true) instead of a plain truncating write, so
it fails atomically (no separate check-then-write race) if a log
already exists for that id -- another run still in progress, or a
finished one never polled/cleaned up. Added GemmaError::RunIdInUse,
mapped to 409 Conflict in the route, caught before any subprocess
runs. Previously a reused id would have silently truncated whatever
was already there.
Verified: the new compute_gemma_rejects_a_run_id_already_in_use test
pre-seeds a log file for an id and confirms the POST is refused (this
needs no gemma-wrapper, since the check runs before any subprocess
call). cargo test passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
/compute/gemma can genuinely take minutes (see the README's ~100K
marker caveat), and until now gave no visibility into a run already
in flight. Same shape GeneNetwork2/3 use for R/qtl
(gn3/computations/streaming.py, gn3/api/streaming.py): the caller
picks a run id, passes it as POST /compute/gemma?id=<id>, and polls
GET /compute/gemma/status/{id}?peak=<offset> on a second connection
while the first is still blocked.
What's actually tailed is gemma-wrapper's stderr, not stdout: under
--json, all of its human-readable progress (banner, chromosome list,
"Executing: ...", per-step info) goes to stderr, and stdout carries
only the one final JSON record, printed once at the very end -- so
there is nothing to stream on stdout. run_streaming() replaces the
old buffered Command::output() call with one that reads stderr line
by line as the process runs, appending each line to a log file if one
was given, across both the kinship and GWA steps into a single
continuous log; stdout is still read in full (unchanged, one JSON
blob). A completion marker ("GN4: computation complete") is appended
once the whole request finishes, the same convention GeneNetwork3
uses ("Done with GN3 Computation" in gn3/api/rqtl2.py) under this
project's own name.
run_id is validated (1-64 ASCII letters/digits/-/_) before it's ever
joined into a path, on both the POST's ?id= and the status endpoint's
path segment -- GeneNetwork3's own /api/stream/{run_id} does no such
check on its equivalent id, which is not knowingly replicated here.
Verified end to end, not just unit tested: the integration test spawns
the POST on its own task and polls the status endpoint from this one
while the ~19s real computation is still running, asserting it
actually observed non-empty progress lines mid-run (not merely after
the fact), then checks the finished log for both real gemma-wrapper
progress text and the completion marker. cargo test passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
commands in the response notes
README: added /compute/gemma to the endpoint table and a dedicated
section with a real example response, and the explicit caveat the
user asked for -- not feasible past roughly 100,000 markers at this
point (synchronous, sequential, no parallel LOCO jobs; the BXD
fixtures, 7,320 markers, already take ~19s, and real datasets go up to
~1M markers).
notes: the response now carries the exact gemma-wrapper invocations
used for the kinship and GWA steps ("Kinship: gemma-wrapper ... -gk",
"GWA: gemma-wrapper ... -lmm 9"), not just a generic description.
kinship_args/gwa_args build the argument list once and feed both the
actual subprocess call and the note text, so the two can't drift
apart. Added unit tests asserting -gk and -lmm never appear together,
and extended the real end-to-end test to check both notes' shape.
Verified: cargo test passes in full, including the real gemma-wrapper/
GEMMA run, with the notes field inspected directly against the actual
command lines used.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
|
|
index; INCREMENTAL crawls, hashes each document's complete source,
replaces only changed documents by identity term, deletes vanished
identity terms, and writes nothing when nothing changed.
* src/indexer/schema.rs: the identity/type/prefix/slot definitions
shared by the writer and the native reader.
* src/indexer/rows.rs: the gene and phenotype crawl. Both queries
carry confidentiality < 1 AND public > 0; the Python phenotype query
has no WHERE clause (the metadata leak this closes by construction).
* src/indexer/build.rs: document construction, mirroring index_genes /
index_phenotypes call for call (TermGenerator configuration, prefix
order, value slots, popped-before-storage fields, identity term).
* src/indexer/pyjson.rs: byte-exact reproduction of Python json.dumps
(separators, ensure_ascii escaping, shortest round-trip float repr
with CPython's half-to-even tie-breaking).
* src/indexer/cache.rs: the RDF RIF/wiki dictionaries, reproducing
build_rdf_cache(remove_common_words=True) from the authoritative
cache file; each dictionary is one monolithic hashed input.
* src/indexer/sha256.rs: dependency-free SHA-256 for the source and
cache hashes.
* src/indexer/sidecar.rs, src/indexer/run.rs: per-document source
hashes in a sidecar next to the index (atomic rename strictly after
the index commit), FULL and INCREMENTAL orchestration, identity-term
uniqueness assertion, and the startup key-set mismatch fallback.
* src/lib.rs: register the indexer module.
* tests/data/pyfloat.txt: reference-docdata float-token fixture for the
serializer test (generated read-only from the reference index).
|
|
GeneNetwork2 (wqflask/marker_regression/gemma_mapping.py) and gn-guile
(gn/runner/gemma.scm) both always use -lmm 9 in their working --loco
invocations. Checked by running it: gemma-gn2 accepts 9 fine, and the
resulting .assoc.txt carries p_lrt, not p_wald -- some mapping other
than stock GEMMA's 1-4 scale, not independently verified here, but
consistent with what both of those codebases rely on in production.
Widened validation from a 1-4 range to the explicit set {1, 2, 3, 4,
9}: 9 is now known-good, but 5-8 remain untested and stock-invalid, so
they're still rejected rather than guessed at.
Updated the real end-to-end test to assert on p_lrt (what lmm=9
actually produces) instead of p_wald, and moved the
out-of-range-lmm test from 9 (now valid) to 5.
Verified: cargo test passes in full, including the real gemma-wrapper/
GEMMA run against the BXD fixtures with the new default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Maps a QTL from a genotype file, a SNP annotation file and trait
values, by running gemma-wrapper's K-matrix step then its LMM
association step, both synchronously (axum/tokio awaits the subprocess
directly, no job queue needed) and returning the parsed .assoc.txt
rows in the standard envelope.
Both steps use a bare --loco (gemma-wrapper derives the chromosome
list from the SNP annotation file). This was arrived at empirically,
not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real
against the BXD fixtures,
- GeneNetwork3's gn3/computations/gemma.py composes a GWA command that
always keeps "-gk" alongside "-lmm"; real GEMMA rejects that
combination outright ("only one of -gk ... -lmm ... options is
allowed"), so none of its gwa-compute/k-gwa-compute endpoints can
produce output. Unsurprising: tests/integration/test_gemma.py is
entirely @unittest.skip, so this was never exercised.
- GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly
omits "-gk" from the GWA step, but its non-LOCO branch fails too,
the same way my own first (non-LOCO) attempt did: gemma-wrapper only
ever archives a non-LOCO kinship matrix into a .tar.xz, with no
loose file at the path its own non-LOCO GWA code assumes
("Enforce failed for file_kin ... open file"). Only --loco, which
gemma-wrapper's own USAGE banner documents and which does unpack
that archive, actually works.
A --loco run's .assoc.txt files (one per chromosome) are themselves
only ever inside gemma-wrapper's own output archive, never loose, so
gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated
dependencies) to extract them before concatenating all chromosomes'
rows into the response.
guix.scm now depends on (gn packages gemma) from guix-bioinformatics
for gemma-wrapper (propagated-input: it's invoked as a subprocess, not
linked, and this makes it land in a consuming profile too, same as
GeneNetwork3's and GeneNetwork2's own packages do it). That dependency
needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see
that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not
build at all before that fix.
Verified end to end: `cargo test` passes in full, including a real
(non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD
fixtures copied from genenetwork3/tests/test_data, producing exactly
7320 association rows (one per marker in BXD_snps.txt) in ~19s.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Add src/embedded.rs with lmdb/duckdb/sqlite round-trip helpers and
tests/embedded.rs integration tests: an LMDB put/get (surviving a
fresh environment handle), an in-memory DuckDB query, and a SQLite
query.
Depend on the system libraries rather than bundled builds:
duckdb 1.10506.0, lmdb 0.8.0 and rusqlite 0.40.2 with their default
features, so the Guix build reuses libduckdb/liblmdb/libsqlite3 instead
of compiling its own.
Guix wiring (guix.scm): add the duckdb, lmdb and sqlite inputs
((gnu packages duckdb)/(gnu packages databases)/(gnu packages sqlite)),
a set-duckdb-paths phase that points libduckdb-sys at the system
libduckdb via DUCKDB_LIB_DIR/DUCKDB_INCLUDE_DIR (there is no
duckdb.pc), and the 79 new crate sources in rust-package.scm
(regenerated with `guix import crate -f Cargo.lock genenetwork4`).
|
|
tied to dev mode.
The endpoint no longer depends on GENENETWORK4_DEV_MODE or on the bind
address. It decides per request whether to answer.
* src/routes/dev.rs (get_test_token): answers a caller only when the TCP
peer (axum ConnectInfo) is in 127.0.x.x (or ::1, or an IPv4-mapped
127.x) and the request carries no X-Forwarded-For, Forwarded or
X-Real-IP header (a reverse proxy on the same host would otherwise make
every client look local). Anything else, including an unknown peer, is a
403. Local is never established from a header.
* src/main.rs: serve with into_make_service_with_connect_info, which
provides the peer address.
* src/config.rs: remove dev_mode, GENENETWORK4_DEV_MODE and
dev_token_enabled(). Add GENENETWORK4_TEST_TOKEN (on by default); when it
is not a clearly affirmative value (true/1/yes/on) the route is not
registered: a 404, absent from /openapi.json. It fails closed, so a
typo ("flase") turns the endpoint off.
* src/lib.rs: register the route unless that switch is off.
* tests: tests/get_test_token.rs rewritten (remote, unknown and proxied
peers refused; 127.x and ::1 answered; always documented by default);
tests/get_test_token_off.rs for the switch (own process: it sets an
environment variable); common::post_from supplies a peer address to the
in-process requests. Python: test_get_test_token.py rewritten, TOKEN_ENV
is now empty (no setting needed).
* README.md, docs/gn-auth.md, docs/SKILL.md, RELEASE_NOTES.md: document
the check, the switch, and "Behind nginx or in a container" (a bridge
container sees the gateway as the peer; a host-network container behind
nginx sees 127.0.0.1, so block the path in nginx, set X-Forwarded-For,
or switch the endpoint off).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
/accessions.
Answers four more questions of aisearch's full_benchmark.csv from
tables the API did not expose.
* src/db.rs (Group, GroupDetail), src/routes/groups.rs: groups gain
`mapping_method`, the name of the MappingMethod row their
mapping_method_id points to (qtlreaper, Rqtl, ...), or null.
* src/db.rs (Dataset): datasets gain `tissue`, the Tissue their
ProbeFreeze points to (e.g. "Brain mRNA").
* src/routes/mapping_methods.rs: GET /mapping-methods lists the
MappingMethod table, including tools no group uses (PLINK, FastMap).
* src/routes/accessions.rs: GET /accessions/{accession} resolves a
GeneNetwork accession number (GN602, gn602 or 602) via InfoFiles to the
dataset's name, title, category (publish/probeset) and group.
* docs/genenetwork_datatypes.md: MappingMethodId is a key into the
MappingMethod table (the authoritative reading); GN2's code-based reading
is noted as superseded.
* tests/benchmark.rs, test/data/benchmark_search.json: add the 60 new
full-benchmark rows that pass (full rows 0..114; search rows run against
recorded production gn3, the rest need the database), and full rows 33
(mapping tools), 34 (Rqtl for outbred sets), 35 (GN602) and 44 (only
GTEx v8 has TPM datasets). The untested rows are listed in the file's
header comment with the reason for each.
* tests, test/python: tests for the new fields and endpoints; both
endpoints added to the OpenAPI and next-link crawl tests.
* README.md, RELEASE_NOTES.md: document the new fields and endpoints.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Each costs a full round trip to the database per pooled checkout (a
ping on take, COM_RESET_CONNECTION on return), on top of the query:
three round trips per database call instead of one. Against a database
about 130 ms away that is 390 ms per call instead of 130 ms. Measured
across 25 SQL endpoints: 1.3x (a 15,060-row list, dominated by row
cost) to 3.0x (single-call endpoints) faster; the full Rust and Python
suites pass.
Nothing here needs the reset (queries are read-only and keep no session
state). The ping only guarded against a connection the server closed
while idle; that now surfaces as a database error on the next query
that uses it (the pool then drops it), without a retry.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Breaking: every success body is now {"data", "meta": {"state",
"took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only
`/` (plain text) and `/openapi.json` are unwrapped.
* src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers
take Ctx first and build the response with ctx.envelope(data), so
timing and the request id are never filled in by hand. `state` is
`complete` or `partial`; notes are server-authored only; `next` are
concrete relative hrefs (path_segment percent-encodes names).
* src/routes/*.rs, src/lib.rs (version): every handler returns
Envelope<..> with next links; /search is `partial` when the locus
lookup fails or a sorted 1000-hit window may be truncated, and
/publications/{pubmed_id}/traits when datasets were left out for
authorization.
* src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member.
* src/auth.rs (AuthToken): the 401 now carries request_id and took_ms
(it never had a request_id); Ctx stores RequestInfo in the request
extensions for it.
* tests: common::data / envelope_data parse every success body through
the envelope; tests/envelope.rs and test/python/test_envelope.py check
the shape, errors, and that every `next` link resolves;
tests/envelope_partial.rs checks the `partial` search without a
database; tests/openapi.rs fails if a JSON success response is not an
Envelope_* schema.
* README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document
the envelope and took_ms on errors.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Composes the new main above the search engine seam:
* src/routes/search.rs (page): parsed and threaded through the seam;
the gn3 forwarder passes page to gn3 unsorted (sorted fetches the
wide first page), the native engine takes the mset window. Ranked
pages slice the sorted list, as before.
* src/routes/search.rs (add_publish_annotations): enrichment now
fills locus and abbreviation in one PublishXRef/Phenotype lookup.
* src/search/mod.rs (Engine::search): page parameter; SearchHit
gains abbreviation and pubmed_id (gn3's future fields flow via
extra, as before).
* src/db.rs (LRS_PER_LOD): 4.604 — lod shifts accordingly everywhere.
All 34 test binaries green; parity replay and shape gates pass
against the dev index with the new lod constant (grading compares
identifier sets, not lod); page disjointness, abbreviation and locus
verified live (native engine, dev index, SQL_URI set).
|
|
Lists the published traits (across groups) a publication reports.
Datasets the caller may not view are left out, using the same check as
GET /traits/{dataset} (traits.rs is_authorised is now pub(crate)).
* src/db.rs (TraitInfo, trait_info_sql): TraitInfo gains `dataset`,
`abbreviation` (Phenotype.Post_publication_abbreviation, else Pre) and
`url` (the GeneNetwork show_trait page). The three TraitInfo queries
now share one SELECT. New get_group_traits_by_abbreviation and
get_publication_traits.
* src/routes/groups.rs (group_traits): add ?abbreviation= filter.
* src/db.rs (Group), src/routes/groups.rs (groups): groups carry
`family` and `species`; add family, species and q filters.
* src/routes/search.rs (search): add `page` (gn3's page unsorted, a
page of the ranked list with `sort`); declare `pubmed_id`; hits gain
`abbreviation` next to `locus` (get_publish_annotations).
* src/db.rs (LRS_PER_LOD): LOD = LRS / 4.604, the divisor the
benchmark's RDF data uses (was 4.61).
* tests, test/python, test/data/benchmark_search.json, README.md: cover
all of the above; tests/benchmark.rs now also answers rows 11, 16, 25,
30, 34 and 37 of small_benchmark.csv, and compares LOD exactly.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
review F1)
The C3.6 guard test set XAPIAN_DB_PATH="" to "prove" the guard fires
before the index is opened — but the error VARIANT is already that
proof (an unset path would surface Internal, an unopenable one
Unreachable, if the query ever reached the index), and the mutation
raced the config tests that set the same key on their own parallel
threads in the lib test binary (a scheduler-dependent flake of the
config assertions, found in the mandatory P3 hostile re-read).
The test now touches no environment at all.
Gate: hermetic lib gate green after the change
(logs/p3-f1-fix-lib.log: LIB_EXIT=0, LIB_FMT_EXIT=0).
|
|
One commit, both halves of AMENDMENT E / the 2026-10-06 contract:
(i) The 422 invalid_query body gains the blessed typed shape. The
SearchError::InvalidQuery payload becomes {message, xapian_error_type},
threaded from every producer:
- native parser rejects (parse_query_checked): the ParseError's own
class, e.g. QueryParserError for Q51/Q52;
- the C3.4 numeric-range guard: QueryParserError ("Unknown range
operation", Q52's recorded production diagnostic);
- the C3.6 invalid-location guard: InvalidOperationError — the class
production actually threw for Q42;
- the gn3 forwarder: gn3's error_type field, with the bytes-repr leak
normalized (b'QueryParserError' -> QueryParserError — the binding-
report R1 / OPEN-16 wrapper-layer normalization), or the exception
class name itself for gn3's escaping-exception 500 bodies.
The route's engine_error builds the typed body inside the existing
ApiError envelope, under "error": {type: "invalid_query", message,
param: "query", detail: {xapian_error_type}} — the envelope's existing
fields (code, detail, ...) keep their meaning, so every current 422
(Q51/Q52 parser rejects, Q42's invalid-location class, mean:abc..def)
emits it and the harness's body["code"] grading is untouched.
The forwarder additionally maps gn3's malformed-processor-value 500s
(the escaping xapian exceptions — Q42's recorded
{"error": "InvalidOperationError", ...} body, the parity-metric
amendment's curl-verified ValueError) onto InvalidQuery, scoped to the
exception-class shape (a bare CamelCase ...Error name, or "Class:
message"): the blessed contract folds all malformed input into 422,
while prose 500 diagnostics ("gn3 exploded") keep the 502
backend_error class — pinned by both the rust and python route tests.
(ii) The harness comparator's Q42 mapping rule changes: recorded
production 500 -> expected 422 invalid_query. The recorded fixture
itself stays untouched as production truth (recorded_status 500,
served verbatim by the stub); only grade_error_class's Server500 arm
re-baselines, with its doc comments and the comparator unit tests.
Gate evidence (~/spikes/port-execution-r2/logs/c37-*): production
floor 54/54 PASS with the NEW rule — Q42 rec=500 gn4=422
code=invalid_query through the real route + forwarding engine + stub
(match200 49/49, clause-b 36/36, min J 1.000, parity-report.json
floor PASS); FULL pinned cargo test --release EXIT=0 (30 suites,
incl. the re-based comparator unit tests and the route unit tests
pinning the typed body's presence for invalid_query and absence for
backend errors); shape-class gate 52/52 unchanged; hermetic lib 23
green incl. the typed-payload assertions (Q51 message byte-identical,
QueryParserError classification). rustfmt --edition 2024 clean on all
nine touched files.
|
|
(C3.6)
gn3's species-location shorthands (search.py:246-272), WITHOUT the Hs
liftover OR-branch (OPEN-4, owner-decided; a C3.6b commit would add
it): `Hs:chr<N>:<pos>` / `Mm:...` expand to the gn3 triplet (search.py:
185-193) — AND of the species boolean term (XShuman / XSmouse), the
chromosome term (XC + chromosome number, value lowercased before the
split exactly like gn3's split_query, search.py:178-183) and the slot-2
position range (the C3.5 machinery: SI suffixes, point +/-50 kb clamp,
one-sided VALUE_GE/VALUE_LE, bases->megabases).
THE INVALID LEAF (Q42, AMENDMENT E): every condition where gn3 raises
ValueError (search.py:179-180, 197-201) — a value not splitting into
exactly chr<chromosome>:<position-spec>, a chromosome not starting
with "chr", or an unparseable position spec — emits the sentinel term
GN4InvalidLocationLeaf (verified absent from the dev index; untypeable
as free text under STEM_ALL_Z). The engine then detects the sentinel in
the parsed query's unique terms and reports SearchError::InvalidQuery
BEFORE the index is opened and any mset call — the blessed 422 for the
recorded production mset crash (Q42's 500 InvalidOperationError), via
the app-level guard (evidence-backed option (b), PORT-DESIGN C3.6,
OPEN-31 closed by AMENDMENT E). The C++ try/catch shim is deliberately
NOT extended; no crash-mimicry. An invalid leaf anywhere in the tree
poisons the whole query, exactly as gn3's mset-time OP_INVALID
rejection did (search.py:293-301 has no try/except).
Gate evidence (~/spikes/port-execution-r2/logs/c36-*): hermetic pins
green (23 lib tests: Q40/Q41 triplet forms XShuman+XC2 / XSmouse+XC1 +
slot-2 VALUE_RANGE, value lowercasing, the 7-case invalid matrix,
OR-combined poisoning, and the guard firing InvalidQuery with NO index
configured — gn3's parse-before-open order preserved); shape-CLASS
gate 52/52 cumulative vs the dev index (Q40/Q41 parse Ok with dev
count 0 — REFERENCE ONLY per AMENDMENT B; probe evidence the triplet
machinery is live: species:mouse AND chr:1 AND mb:1..200 matches, the
4.95..5.05 Mb chr1 window is genuinely empty in the 2025-02 dev data;
Q42 -> InvalidQuery); route-level scenarios: Q52 AND Q42 both
422 invalid_query through the real app with SEARCH_ENGINE=native.
rustfmt --edition 2024 clean on the touched files.
|
|
The gn3 SI-suffix FieldProcessor on slot 2 (search.py:237-241), for both
the `position` prefix and its `pos` alias, sharing one processor type
(each registration Box::leaks its own instance inside the binding; the
one-per-process parser absorbs that).
Ported faithfully from search.py:136-165:
- apply_si_suffix (136-142): trailing k/m/g (case-insensitive) multiply
by 10^3/6/9; an UNSUFFIXED value must be an integer (`int(location)`)
— `position:1.5` declines exactly where gn3 raises ValueError.
- parse_position (145-155): a `..` spec is a range with either side
optionally empty; a point expands to +/-50 000 bases clamped at 0.
- the slot-2 query (158-165): sides convert bases->megabases and keep
gn3's one-sided C++ NumberRangeProcessor semantics (VALUE_GE /
VALUE_LE for an empty side, measured identical to the bridge forms);
the degenerate both-empty range is MatchNothing (binding-report R3's
documented ("","") divergence, no production shape).
AMENDMENT A verified end-to-end: the fork's decline->MatchNothing bridge
(aa915af) is IN the dependency and behaves as documented — an
unparseable spec (`position:abc`) parses to an empty-match query and
serves 200 with 0 hits (POSABC gate row, dev index), never an <INVALID>
leaf, never a process abort; `position:abc OR alcohol` keeps its real
OR arm (xapian documents MatchNothing as boolean false). The
registration-only job was the lane's plan; no re-fixing of the library.
Q53 `position:` never reaches the processor (empty value -> free text,
recorded production 200 with 576 hits; dev reference count printed).
Gate evidence (~/spikes/port-execution-r2/logs/c35-*): hermetic pins
green (21 lib tests — slot-2 VALUE_RANGE/VALUE_GE/VALUE_LE forms, SI
case-insensitivity, point clamp, decline matrix incl. MatchNothing OR
semantics); shape-CLASS gate 49/49 cumulative vs the dev index
(Q32=1, Q33=2, Q34=1 dev gene hits at 1.05-1.06Mb/7.45-7.55Mb, Q53=1017
free-text hits, POSABC=0 — dev counts REFERENCE ONLY per AMENDMENT B).
rustfmt --edition 2024 clean on the touched files.
|
|
(C3.4)
NumberRangeProcessor on mean (slot 0), peak (1), peakmb (3), additive
(4), year (5) per search.py:234-243 (slots verified literal at
index-genenetwork:293-299); the markers carry the colon exactly like
gn3's `NumberRangeProcessor(i, prefix + ":")`.
The fork's processor parses bounds as f64 (commit 2673115, Amendment A:
the C3.4 "apply the f64 fix" workload is done at the library level) and
its bridge maps empty sides onto gn3's one-sided VALUE_LE / VALUE_GE
semantics natively — measured on the pinned toolchain before coding
(probe/, logs): `mean:10..20` -> VALUE_RANGE 0, `mean:..15` ->
VALUE_LE 0, `mean:10..` -> VALUE_GE 0. The skeleton-era 0.0/1e18
substitution is therefore superseded for these registrations by the
strictly more faithful one-sided forms.
AMENDMENT A nuance measured and handled: the fork's decline->MatchNothing
bridge (aa915af) also swallows NON-NUMERIC range bounds at the parser
level (`mean:abc..def` parses Ok to MatchNothing — probe evidence), so
the recorded production 400 "Unknown range operation" (Q52, graded 422
invalid_query) can no longer come from the parser. The engine therefore
re-checks the C++ NumberRangeProcessor contract itself (guard_numeric_
ranges): unquoted `<prefix>:` runs containing ".." must have f64-parse-
able non-empty sides, else SearchError::InvalidQuery("Unknown range
operation") before the index is opened. Residuals documented in the
guard's doc comment (inf/nan forms, the degenerate both-empty range —
binding-report R3's ("","") divergence, no production shape).
Gate evidence (~/spikes/port-execution-r2/logs/c34-*): hermetic lib pins
green incl. per-slot VALUE_RANGE/VALUE_LE/VALUE_GE parse forms (a
wrong-slot registration goes red) and the guard accept/reject matrix;
shape-CLASS gate 44/44 vs the dev index (Q28-Q31,Q35-Q39,Q44 parse Ok
with dev counts printed as REFERENCE ONLY per AMENDMENT B: mean:10..20
=1741, mean:..15=23132, peak:5..15=21857, year:2010..2020=22081, Q44
=332; Q52 -> InvalidQuery "Unknown range operation"); wide-range probe
proves slots 0/3/4/5 live on the dev index so the 0-hit shapes are
data-truth; new route-level native scenarios (tests/search_native_
route.rs) pin Q52's STATUS CLASS at the route: 422 invalid_query, with
a valid range serving 200. rustfmt --edition 2024 clean on every file
this commit touches.
|
|
native-search-engine.
Resolution: his route-level additions compose above the engine seam —
the handler now validates sort/order, asks the engine for
SORT_FETCH_LIMIT (1000) when ranking, maps SearchError via the seam's
engine_error, then computes lod, sorts, truncates to limit and fills
loci; lod/locus join SearchHit in src/search/mod.rs. His tests pass
through the seam.
KNOWN at this commit: the parity floor is BLOCKED (36 shapes) — the
route's new truncate(limit) exposes that route-level full-set grading
was a calibration-stub artifact (real gn3 caps at per_page; the stub
ignored it). Harness fix queued: set-equality moves to engine level,
status + top-10 stay route level. Landed together with that fix.
|