| Age | Commit message (Collapse) | Author |
|
Asked to map a 9-strain trait, the wrapper scanned it against the
6-individual fixture cross and returned six markers with LOD scores and
`state: complete` -- a plausible answer belonging to nobody. R recycled
and truncated the values silently, and nothing downstream could tell.
GeneNetwork3's copy of this script has the same hole.
The cross-file format carries no per-sample names, so the count is the
only check available; order remains the caller's responsibility (and the
reason values and genotype file must come from the same panel). With the
check, the same request is a 502 rqtl_failed quoting both counts, and the
correctly-shaped request still returns the documented LODs.
* scripts/rqtl_wrapper.R: compare the pheno file's rows with nind() of
the cross, and stop.
* README.md (/compute/rqtl): the second fix relative to gn3, and what
stays the caller's job.
* docs/www/ai-instruction.md (4a): values and genotypes must describe the
same individuals, one for one, in the same order.
* docs/www/operator-checklist.md (Compute): what genotype files actually
exist on this box (test fixtures, 198 BXD individuals) and why a database
trait, with its own strain subset, cannot be mapped against them.
|
|
Fetching one trait's value per strain is usually the step before mapping
that trait, and the API can do the mapping. Those calls are POSTs with a
body, so they cannot go in `next` -- a `next` link is a GET that returns
200, which is what makes `next` safe to follow blindly and what
tests/envelope.rs relies on. Hence a fifth envelope member, `actions`,
holding method + href + description + body, and omitted entirely when
there is nothing to suggest (most responses keep the four-member shape).
The three per-trait values endpoints now carry a `POST /compute/rqtl`
and a `POST /compute/gemma` suggestion with the values already in the
body, in the response's own sample order. Genotype/SNP paths stay
bracketed placeholders: they must exist on the server, which the server
cannot know on the caller's behalf.
The instruction page says, in the same breath, never to run R/qtl or
GEMMA locally -- a local scan would use genotypes that are not the
panel's and would disagree with GeneNetwork's own results -- and that
results are cached, so repeat calls are cheap (measured: ~2 s cached
against ~20 s cold on the BXD fixtures).
Also fixes a flake the new test exposed rather than caused:
config::tests read and write process-global env vars and Config::load()
reads all of them, so the RAVANAN_* tests could fail
search_engine_keys_follow_env_over_default_layering on an unrelated
assertion. One module-level lock serialises them.
* src/response.rs (SuggestedCall, Envelope::actions/with_action/
with_actions): the new member, omitted when empty; unit tests for both.
* src/suggest.rs: new. Builds the two suggestions for either value type
(db::TraitValue, db::ProbeSetValue), with its own unit tests.
* src/routes/{traits,groups,datasets}.rs: the three values handlers.
* tests/compute_suggestions.rs: new. Suggestions present with values in
order, on all three routes; absent elsewhere; `next` still GET-only and
still returns 200.
* src/config.rs (tests): env_guard, taken by the 11 config tests.
* README.md (Success responses): the `actions` member, with an example.
* docs/www/ai-instruction.md (4a): never compute locally, why, caching,
and how to use the suggested bodies.
* docs/www/operator-checklist.md: the caching numbers.
|
|
R/qtl and GEMMA are part of the service now, so the blanket 403 goes:
the handlers already require an AuthToken, which is the whole gate
("token_only", not resource-checked). What that buys and costs is now
written down rather than implied -- any token holder can name
server-side paths for R and gemma to read, and the public-data-only
restriction on traits/datasets does not apply here at all.
The box also has to be able to run them: PATH must carry Rscript,
gemma-wrapper and gemma, *and* R_LIBS_SITE must point at the profile's
site-library or scripts/rqtl_wrapper.R dies on `there is no package
called 'optparse'` -- which reads like a token or proxy fault and is
neither.
Verified through nginx with a token: POST /gn4/compute/rqtl 200 in 0.4s
with the fixture's marker rows, POST /gn4/compute/gemma 200 in 2.1s with
7 320 marker rows (identical to the documented run's first row),
GET /gn4/compute/gemma/status/<id> ending in "GN4: computation
complete", /compute/correlation 200; all four 401 without a token.
* etc/nginx-vesuvio.pu.ac.ke.conf: drop the /compute 403 locations, keep
them commented as the way back; timeouts/body-size comments now apply.
* docs/www/ai-instruction.md: new §4a, "Running QTL mapping (compute)" --
shapes, server-side paths, ?id= + status polling, what to do with a 504.
* docs/www/operator-checklist.md: compute moves from "must be off" to
"proxied, know what it grants", plus what the box needs (PATH,
R_LIBS_SITE, ravanan) and the measured results.
* docs/nginx.md: same, in the deployment and checking sections.
|
|
The instruction page is now also the way to get the rest of the docs:
clone the repo and read docs/. Measured, so the numbers are real -- a
--depth 1 --sparse clone of docs is ~2.5 MB in a couple of seconds; the
plain --depth 1 clone is 17 MB, almost all of it source tree the reader
does not need.
Why clone rather than fetch: the forge's tree pages sit behind Anubis (a
browser proof-of-work challenge) and its raw endpoint answers 403, so a
plain HTTPS fetch of a Markdown file returns an HTML challenge page. The
git smart-HTTP endpoints pass, so clone works -- which is what makes
this instruction usable at all.
Also says how to tell whether the docs match the running build
(`git log -1 --format=%h` against the `git` field of /gn4/version).
* docs/www/ai-instruction.md: "Get this document, and the rest of the
docs, by cloning", ahead of the numbered sections.
|
|
The AI instructions described the field prefixes but not the semantics,
so a reader would reasonably assume AND between words, working quoted
phrases and wildcard prefixes -- none of which is true here. Measuring
against the production index: juxtaposition is OR (`alcohol` 1 200 vs
`alcohol AND dependence` 45), free text is stemmed and
case-insensitive, quoted phrases and NEAR/ADJ match *nothing* because
the stemmed terms carry no positions (a hyphenated word is a phrase, so
`insulin-like` also matches nothing), and wildcards are not enabled
(`alcohol*` is just `alcohol`).
* docs/www/ai-instruction.md (3. Search): rewritten around the Xapian
QueryParser semantics and a table of exact / stemmed / numeric-range
fields.
* docs/search-engines.md: the same, for whoever maintains the engine,
plus the gn3-forward-vs-native difference in what counts as a field.
|
|
The page said a human issues tokens and to "stop and ask", without
saying whose door to knock on -- and the operator commands belong in the
operator file anyway. State it as the GeneNetwork team, with wording the
assistant can relay to its user: what a token is for, how long the
default lasts, and to pass it out of band rather than through a public
issue or a shared chat. Same wording for renewal, since a 401 is the one
failure the assistant cannot fix itself.
* docs/www/ai-instruction.md (1. Get a token): "If you do not have one"
and "If it stops working", the latter quoting request_id and retrying
exactly once; the summary line and the 401 row of the error table say
GeneNetwork team rather than "the operator".
* docs/www/operator-checklist.md (Token issuance): the request path, so
the team knows what the ask they receive looks like.
* .gitignore: *.swp, after one got swept into this commit.
|
|
ai-instruction.md was written before anything was deployed: it took a
placeholder base URL, did not say where the API is mounted, and buried
the operator's deployment notes in an appendix labelled "not for the AI
reader" -- awkward to hand to an assistant, and confusing when it is.
So: the AI-facing page now names the deployment
(https://vesuvio.pu.ac.ke/gn4) and is self-contained, and the operator
material moves to operator-checklist.md. Everything in the page was
exercised against the running deployment.
Content changes, beyond the move:
* `next` hrefs arrive already prefixed (`/gn4/groups/BXD/strains`), so
they resolve against the origin -- joining them to the current URL's
directory double-prefixes them. Said plainly, since it is the one
thing the prefix breaks.
* says what cannot work and should not be retried: get-test-token is a
404, /compute/* is a 403 from the front end, and neither is a
permissions problem to work around.
* `/publications/{pubmed_id}/traits` wants the PubMed id, not the
internal `publication_id` in the same trait record, which 404s.
* states the current rate-limit position (none configured) instead of
implying a 429 policy that does not exist yet.
* docs/www/ai-instruction.md: rewritten, paste-ready.
* docs/www/operator-checklist.md: new file, the former appendix, updated
with what is now enforced at the proxy and a record of the verified
request matrix.
* README.md, docs/nginx.md, etc/nginx-vesuvio.pu.ac.ke.conf: point at
the operator checklist by its new name.
|
|
Without it /search forwards to GeneNetwork3 at GN3_URL (:5003) and
answers 502 backend_unreachable when nothing is listening there -- a
correct error, with the error body's links rewritten, but easy to
misread as the proxy being wrong. The deployment has an index, so native
is what it wants.
* docs/nginx.md (The server behind nginx): add the variable and say what
the default does.
* etc/nginx-vesuvio.pu.ac.ke.conf: same, in the start-up comment.
|
|
Reproduced from a mirror of the deployed /etc/nginx: with
sites-enabled/default still present, nginx logs
conflicting server name "vesuvio.pu.ac.ke" on 0.0.0.0:443, ignored
and answers 443 with the default site -- stock welcome page at /,
nginx's own 404 for /gn4/..., HTTP/1.1 despite `http2 on` in this block.
sites-enabled/* is globbed alphabetically and the first block whose
server_name matches wins, so `default` (edited by certbot to serve this
hostname) shadows the gn4 block. Removing it gives the expected 302 /
301 / 200 / 404 / 403 matrix.
* docs/nginx.md (nginx): say to install the snippet instead of the
distribution default, and what the shadowing looks like from outside.
|
|
The certificate on :443 carries subjectAltName DNS:vesuvio.pu.ac.ke, and
vesuvio.pu.ac.uk does not resolve at all, so the config as committed
could not have matched a request for the real name: nginx falls through
to the default server, which is why /gn4/... answered 404 from nginx
rather than a 502 from genenetwork4.
* etc/nginx-vesuvio.pu.ac.uk.conf -> etc/nginx-vesuvio.pu.ac.ke.conf:
server_name and the certificate paths.
* docs/nginx.md: the same rename, and a "Checking it" section that
starts from what actually gets in the way on the box -- TLS validates
the URL hostname rather than a Host: header, so on-box checks need
--resolve against the name on the certificate, and a mismatch there and
in server_name look identical (default server answers).
|
|
docs/nginx.md explained the pieces; this is the assembled file for the
deployment actually being set up, so it can be reviewed and installed
without stitching anything together. It lives in etc/ rather than
docs/, being deployable configuration rather than prose.
It refuses /compute/*, which the README-based template did not: those
endpoints take server-side file paths and run gemma/ravanan/R on them,
so any token holder able to reach them can read the box -- the operator
checklist in docs/www/ai-instruction.md already said to keep them on the
internal listener, and this is that rule where it belongs.
* etc/nginx-vesuvio.pu.ac.uk.conf: new file. TLS, http2, log format
carrying x-request-id, / and /gn4 redirects, proxy_pass with the
prefix-stripping trailing slash, raised timeouts, sub_filter rewriting
of href/next/spec_url/type, and 403/404 refusals for /compute/ and
/get-test-token.
* docs/nginx.md: point at it, and say why /compute/ is refused.
|
|
|
|
The page told the operator to mint with POST /get-test-token, which is
gn-auth's dev password grant and does not exist in the deployment being
set up. Issuing a second user's token also had no good answer: the stub
minted exactly one token per process, so a new user meant starting a
throwaway instance, grabbing --token-file and killing it.
So add --mint-only: mint for --subject, print, write --token-file and
exit without listening. Verification only depends on the key (committed,
so shared by every instance) and on iss, which now falls back to
AUTH_SERVER_URL -- a token minted this way verifies against the
long-running stub untouched.
* scripts/gn-auth-stub.py (main): --mint-only; --issuer defaults to
AUTH_SERVER_URL before the port-derived default. Docstring says how a
second user's token is issued.
* docs/www/ai-instruction.md (1. Get a token): the operator's command is
the stub, not /get-test-token; the token lasts whatever --expires-in said
rather than an hour, and there is no early revocation. (4. What is
public): minting is operator-side. Appendix: "Token issuance" rewritten
around the stub -- public-data-only, expiry as the only revocation, sub
as attribution, one token per user -- with gn-auth kept as what replaces
it later; the stub is no longer listed as a dev leftover to delete, but
as the auth server that must never be routed publicly.
* README.md (Tests): mention --mint-only next to the stub's other flags.
|
|
The deployment being set up is https://vesuvio.pu.ac.uk/gn4 proxying to
a loopback port, with no gn-auth yet. Two things there are not obvious
and neither is discoverable from the README's /get-test-token section:
every link the API emits is root-relative, so under /gn4 they resolve
outside the mount point; and /compute/* runs for minutes in a single
request, so nginx's one-minute default read timeout cuts it off
mid-computation.
The link rewriting is the delicate part -- the server's serde output is
compact ("href":"/"), not the pretty-printed shape the README examples
show, so a filter written from those examples silently does nothing.
Both spellings of all four affected keys (href, next, spec_url, type)
are in the config, and the envelope's "next":[ array is deliberately
left alone.
* docs/nginx.md: new file. Server environment, the nginx block, what the
prefix does to links and the three ways to handle it, stub-minted
tokens instead of gn-auth, and how to check the result.
* README.md (Behind nginx or in a container): point at it.
|
|
docs/www/ai-instruction.md is the AI-facing page: you are handed a JWT
(the operator mints it on the server's loopback interface, so there is no
token endpoint on the public host and no flow for the model to follow),
send it as a bearer token, and read the envelope. It covers discovery
(/openapi.json, /version), the readable endpoints, search's grammar and
paging cost, the public-data rule (public > 0 && confidentiality < 1, or
older than five years), error classes and what to do with each, and a
worked example.
The operator appendix records the deployment findings from this session:
* /get-test-token is now the minting route and only a loopback peer
check plus proxy-header refusal keeps it off the internet -- it must
not be proxied and the app must stay bound to 127.0.0.1;
* POST /compute/* takes server-side file paths and is token_only, so it
is an arbitrary-read/execution surface and stays internal;
* the public rule is enforced per dataset for values, while metadata
endpoints are token_only by design and do reveal non-public dataset
records -- a decision to make, not an accident;
* the index must be built with the public filter (GN4's indexer does;
gn3's index-genenetwork has no WHERE clause). Verified on the current
full index: the confidentiality=1 HCPPublish and the public=0
BR_U_0303_M dataset both return zero hits;
* error detail can carry the database address, and tokens minted this way
all carry gn-auth's development user as sub, so the audit trail is
"someone with a token" until client_credentials exists.
|
|
The index stores every boolean term in lowercase and holds no
mixed-case variants at all: XDSbxdpublish (15 060 docs), Awilliams
(10 958), XSmouse, XCx, XYbdnf, XImrna all exist, while XDSBXDPublish,
AWilliams, XSMouse, XCX, XYBDNF and XIMRNA report "not in database" on
both the dev index and a full production one. Matching the value
exactly therefore rejected the way a caller naturally writes it --
dataset:BXDPublish returned 0 hits against an index holding 15 060
BXDPublish traits, author:Williams 0 against 10 958 williams authors.
Lowercase the value for every boolean prefix, generalising the
group-only rule gn3 already has (search.py:204-209). No graded parity
shape changes: they all use lowercase values already.
Pinned per prefix in boolean_prefixes_parse_to_exact_index_terms, so a
registration that regresses to an exact add_boolean_prefix fails there
instead of silently returning zero hits.
|
|
Measured the same request (BXD fixtures, three chromosomes) through
/compute/gemma, ravanan's single-machine backend and ravanan on Slurm,
cold and warm. The Slurm backend is dispatch-bound at this size -- one
scheduler tick per step, with each job itself running in under a second
-- so cold and warm are identical there, and the README's 16.7s figure
belongs to the single-machine backend.
Also document the Slurm-side requirement behind that measurement: a
completed job has to stay queryable for as long as the workflow runs.
Once slurmctld purges the record ravanan falls back to slurmdbd, and on
a cluster without accounting the workflow aborts on a 404 long after the
job succeeded.
|
|
cwl/README.md explains the whole path (API -> ccwl compile -> ravanan ->
slurmrestd -> slurmd), where Guix sits (front end only: nodes just need
the same /gnu/store mounted, plus bash, a writable ravanan store for
data/logs, and scratch for the job's working directory), the two caches
(ravanan's work reuse, checked before submitting; gemma-wrapper's
per-chromosome --cache-dir, inside the job) and why the latter is what
actually saves work for this endpoint.
It also answers "can the nodes rebuild the tools instead of sharing a
store": yes in principle, since Guix is deterministic and a node
building the same derivations would materialise identical store paths,
but ravanan submits a store path and Slurm does not substitute
automatically -- so it needs a realise-then-run wrapper (ravanan's
slurm backend already holds the script's derivation), a per-workflow
realise job, or guix publish plus a node-side daemon, with the costs
and channel-pinning caveats listed. The timing table from the test
cluster (cached: identical; fresh: waves vs synchronous steps) is
included so expectations are explicit.
docs/ravanan-slurm.md and README.md link to it.
|
|
POST /compute/gemma/ravanan used to run every CWL step locally, because
compute_ravanan never passed ravanan a batch system. Add a
RavananBatch setting:
- RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing
closed on anything else, exactly like GN4_DB_MODE;
- slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint,
--slurm-jwt (a *file*, which is what ravanan reads) and, when set,
--slurm-partition/--slurm-nice; a slurm selection without an endpoint
or a JWT file refuses to start rather than 502-ing every request;
- --scratch defaults to ravanan_cache_dir for slurm, so the store and
the job scratch space live on the same shared filesystem (with slurm
that directory has to be visible to the compute nodes);
- the response's ravanan note records which backend ran.
The switch building is RavananBatch::ravanan_args so it is unit
testable; compute_ravanan takes the backend and the scratch directory,
and the route passes them from config. docs/ravanan-slurm.md documents
the keys, the shared-directory requirement, and the slurm-side auth
wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in
slurmrestd's environment) that makes REST-submitted jobs actually run.
|
|
Realigns every pipe-table column to a uniform width, and trims cell
text so each table stays under 100 chars per row, moving one
over-long cell's detail (InbredSet.MappingMethodId) into a footnote
paragraph below its table instead of dropping it. The gn-auth.md
endpoints table is left over 100 chars, with a note explaining why: its
longest path is 66 chars by itself, leaving no room for the other two
columns under that cap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Adds a gnauth-macros proc-macro crate and a private-field Authorized
capability type in src/auth.rs, so #[gnauth(secured)] is a real,
compile-time-checked guarantee that a handler performs the gn-auth
resource-authorisation round trip before sharing trait/probeset
values, rather than just a comment. #[gnauth(token_only)] marks
handlers that require a token but skip the round trip, and
#[gnauth(disabled)] marks the two handlers that need no token at all.
Also fixes groups.rs: group_trait_values, group_trait_info and
group_traits previously performed no gn-auth round trip at all for
published-trait data; they now resolve the group's PublishFreeze row
and call the same auth::check used elsewhere, denying by default when
no such row exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
tied to dev mode.
The endpoint no longer depends on GENENETWORK4_DEV_MODE or on the bind
address. It decides per request whether to answer.
* src/routes/dev.rs (get_test_token): answers a caller only when the TCP
peer (axum ConnectInfo) is in 127.0.x.x (or ::1, or an IPv4-mapped
127.x) and the request carries no X-Forwarded-For, Forwarded or
X-Real-IP header (a reverse proxy on the same host would otherwise make
every client look local). Anything else, including an unknown peer, is a
403. Local is never established from a header.
* src/main.rs: serve with into_make_service_with_connect_info, which
provides the peer address.
* src/config.rs: remove dev_mode, GENENETWORK4_DEV_MODE and
dev_token_enabled(). Add GENENETWORK4_TEST_TOKEN (on by default); when it
is not a clearly affirmative value (true/1/yes/on) the route is not
registered: a 404, absent from /openapi.json. It fails closed, so a
typo ("flase") turns the endpoint off.
* src/lib.rs: register the route unless that switch is off.
* tests: tests/get_test_token.rs rewritten (remote, unknown and proxied
peers refused; 127.x and ::1 answered; always documented by default);
tests/get_test_token_off.rs for the switch (own process: it sets an
environment variable); common::post_from supplies a peer address to the
in-process requests. Python: test_get_test_token.py rewritten, TOKEN_ENV
is now empty (no setting needed).
* README.md, docs/gn-auth.md, docs/SKILL.md, RELEASE_NOTES.md: document
the check, the switch, and "Behind nginx or in a container" (a bridge
container sees the gateway as the peer; a host-network container behind
nginx sees 127.0.0.1, so block the path in nginx, set X-Forwarded-For,
or switch the endpoint off).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
/accessions.
Answers four more questions of aisearch's full_benchmark.csv from
tables the API did not expose.
* src/db.rs (Group, GroupDetail), src/routes/groups.rs: groups gain
`mapping_method`, the name of the MappingMethod row their
mapping_method_id points to (qtlreaper, Rqtl, ...), or null.
* src/db.rs (Dataset): datasets gain `tissue`, the Tissue their
ProbeFreeze points to (e.g. "Brain mRNA").
* src/routes/mapping_methods.rs: GET /mapping-methods lists the
MappingMethod table, including tools no group uses (PLINK, FastMap).
* src/routes/accessions.rs: GET /accessions/{accession} resolves a
GeneNetwork accession number (GN602, gn602 or 602) via InfoFiles to the
dataset's name, title, category (publish/probeset) and group.
* docs/genenetwork_datatypes.md: MappingMethodId is a key into the
MappingMethod table (the authoritative reading); GN2's code-based reading
is noted as superseded.
* tests/benchmark.rs, test/data/benchmark_search.json: add the 60 new
full-benchmark rows that pass (full rows 0..114; search rows run against
recorded production gn3, the rest need the database), and full rows 33
(mapping tools), 34 (Rqtl for outbred sets), 35 (GN602) and 44 (only
GTEx v8 has TPM datasets). The untested rows are listed in the file's
header comment with the reason for each.
* tests, test/python: tests for the new fields and endpoints; both
endpoints added to the OpenAPI and next-link crawl tests.
* README.md, RELEASE_NOTES.md: document the new fields and endpoints.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Breaking: every success body is now {"data", "meta": {"state",
"took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only
`/` (plain text) and `/openapi.json` are unwrapped.
* src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers
take Ctx first and build the response with ctx.envelope(data), so
timing and the request id are never filled in by hand. `state` is
`complete` or `partial`; notes are server-authored only; `next` are
concrete relative hrefs (path_segment percent-encodes names).
* src/routes/*.rs, src/lib.rs (version): every handler returns
Envelope<..> with next links; /search is `partial` when the locus
lookup fails or a sorted 1000-hit window may be truncated, and
/publications/{pubmed_id}/traits when datasets were left out for
authorization.
* src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member.
* src/auth.rs (AuthToken): the 401 now carries request_id and took_ms
(it never had a request_id); Ctx stores RequestInfo in the request
extensions for it.
* tests: common::data / envelope_data parse every success body through
the envelope; tests/envelope.rs and test/python/test_envelope.py check
the shape, errors, and that every `next` link resolves;
tests/envelope_partial.rs checks the `partial` search without a
database; tests/openapi.rs fails if a JSON success response is not an
Envelope_* schema.
* README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document
the envelope and took_ms on errors.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
-- name the repo so the command is findable from gn4.
|
|
prefix (scripts/index-genenetwork, verified against the click CLI);
plain 'planned response enrichment' instead of naming a proposal;
state the behavior-preserving default up front.
|
|
gn3 proxy by default, the native xapian engine on
SEARCH_ENGINE=native + XAPIAN_DB_PATH); README.md (search): point
to it. House style: scope line, takeaway headings, code pointers,
known quirks (locus null reason, sort tail difference, no runtime
indicator).
|
|
Captures the reusable practices from this session's token-gating work,
for reuse on a similar Rust/Axum service:
- utoipa: a document-wide default security(...) on #[derive(OpenApi)]
instead of repeating it per operation, with per-operation empty
security() overrides for the exceptions; a shared
#[derive(utoipa::IntoResponses)] type for a response shape (e.g. 401)
repeated across many handlers, referenced bare in responses(...).
- Authentication/authorization: an opt-in-per-handler extractor
(AuthToken) over blanket middleware; keeping "is this token valid"
and "is this specific resource public" as independent checks;
resolving a (container, item)-addressed resource's container
category before ever looking up item, rather than guessing item
across categories; a dev-only token-issuing endpoint needing at
least two independent config conditions (not runtime-detected),
registered conditionally rather than registered-and-rejected, with
credentials defaulting to the target service's own published dev
fixtures.
- Testing: a test suite fetching its own token from that dev endpoint
(in-process for an in-process Rust suite, via per-test-class
subprocess env for a spawned-binary Python suite) instead of
requiring a human to provision one; the concurrency hazard of
mutating process-wide env vars across #[tokio::test] functions in one
file, and why a spawned-subprocess-per-class suite doesn't have it;
Rust edition 2024's unsafe env::set_var/remove_var.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs,
TestToken replacing DevToken), matching tests/get_test_token.rs and
test/python/test_get_test_token.py (renamed from
tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and
the GN_AUTH_DEV_* config keys keep their names -- they're about the
dev-token feature generically, not tied to this one route's path.
Every endpoint now requires a valid token, not just the
dataset_probeset*/traits* family: species(_by_name/_datasets),
groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all
gain an AuthToken parameter. /, /version, /openapi.json and
/get-test-token itself are the only exceptions -- a health check, a
version check, the spec describing all this, and the endpoint that
hands out a token in the first place (which would be circular if it
needed one).
Rather than repeating security(("bearer_token" = [])) on every
#[utoipa::path], ApiDoc now declares it once as the document's default
security(...), and the three exceptions besides /get-test-token (/,
/version) override it with their own empty security() -- standard
OpenAPI semantics (an operation's own security, even empty, overrides
the document default; an absent one inherits it). error::Unauthorized,
a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the
shared 401 every gated handler now references bare in its own
responses(...) list instead of writing out the same
(status = 401, description = "...", body = ApiError) tuple by hand.
Verified the resolved security per operation in a live /openapi.json.
tests/common::auth_token() (Rust) and the new
test/python/auth_helpers.auth_token() (Python, deduplicating what four
files previously each defined locally) now try this server's own
POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN --
every previously-ungated test file gains a token check and switches to
get_with_token; the whole suite, both languages, now runs with zero
manually-provisioned tokens given GENENETWORK4_DEV_MODE=true
GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains
EXTRA_ENV so each Python test class can start its own server with dev
mode on. Verified both configured (auto-fetch) and unconfigured
(GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g.
the Guix build sandbox) paths.
README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for
the rename and the new blanket token requirement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Calls gn-auth's own password grant (POST /auth/token) for gn-auth's
well-known development user and returns the result, so local testing
no longer needs a manual curl/gn-auth round trip of its own.
Only ever registered -- not just rejected at request time, genuinely
absent from the router and from /openapi.json -- when two new Config
fields are both set: dev_mode (GENENETWORK4_DEV_MODE) and a loopback
bind (GENENETWORK4_BIND, also new). Both are config, not something
genenetwork4 infers from its actual network exposure, by design: dev
mode alone, or a loopback bind alone, each leave it disabled, and the
check (Config::dev_token_enabled) is a plain function so the condition
itself has direct unit test coverage rather than only being exercised
indirectly through the route.
src/main.rs now binds to config.bind instead of a hardcoded "0.0.0.0",
via a proper SocketAddr rather than string formatting -- the latter
would mishandle an IPv6 bind address like "::1" (needs brackets).
GN_AUTH_DEV_CLIENT_ID/_CLIENT_SECRET/_USERNAME/_PASSWORD default to
gn-auth's own published dev fixtures (flask init-dev-clients,
gn_auth/wsgi.py), so this works unconfigured against an unmodified dev
gn-auth instance -- confirmed live, including using the returned token
against a real gated endpoint (/traits/BXD/10001), not just checking
the token response's shape.
tests/dev_token.rs covers all four combinations of the two settings in
one sequential test function (mutating the same two env vars from more
than one #[tokio::test] in a file would race, since they run
concurrently by default) plus the /openapi.json visibility check.
test/python/test_dev_token.py mirrors this using separate test classes
instead, which is safe here since each spawns its own server
subprocess with its own environment (ServerTestCase gains EXTRA_ENV for
this).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
A new, additional endpoint family that addresses both trait categories
the same way -- (dataset, trait) -- matching GeneNetwork2's own generic
wqflask.api.router.trait_sample_data/get_trait_info, rather than the
two category-specific shapes genenetwork4 already has
(/datasets/{dataset}/probesets... and /groups/{group}/traits...), which
are unchanged and keep working exactly as before.
- db::resolve_trait_dataset(name) resolves a dataset name against
ProbeSetFreeze first, then the new db::PublishDataset (a PublishFreeze
row, via the new get_publish_freeze_by_name) -- whichever matches
decides the category. This means only dataset names are ever
"guessed" across categories, not trait names/ids, which is the much
smaller and safer surface: GN2's own equivalent tries ProbeSet.Name
and PublishXRef.Id against the same bare trait string and silently
falls back, which can return the wrong trait on a same-named
collision. get_publish_freeze_by_name accepts a PublishFreeze's own
Name/ShortName/Id or its owning group's Name/InbredSetName/FullName/
Id, mirroring GN2's own flexible matching (e.g. "BXDPublish" or
"BXD").
- GET /traits/{dataset}: list every trait in a dataset, tagged by
category.
- GET /traits/{dataset}/{trait}: one trait's metadata. A non-numeric
trait against a Publish dataset is a 404 (can't match a
PublishXRef.Id), not a 500.
- GET /traits/{dataset}/{trait}/values: that trait's per-strain values.
TraitValue and ProbeSetValue are already identical in shape, so this
unifies with no information loss, unlike the metadata endpoint above.
Metadata responses are a tagged union ({"category": "probeset"|
"publish", ...}) wrapping ProbeSetInfo/TraitInfo unchanged, rather than
a lossy common-fields schema -- the two share little beyond
mean/locus/lrs/additive. utoipa generates this as a oneOf with a
category discriminator, verified against /openapi.json.
Gated the same way as /datasets/{dataset}/probesets..., extended to
Publish data for the first time: /groups/{group}/traits itself is
still not gated, but reaching the same data through /traits/{group}
now requires a token and checks PublishFreeze.public/confidentiality.
No gn-auth-side change was needed -- POST /auth/data/authorisation
already dispatches *Publish-suffixed dataset names to its phenotype
resource resolver (gn_auth/auth/authorisation/data/phenotypes.py),
the exact mechanism this integration already relies on for mRNA data.
auth::VIEW_RESOURCE_PRIVILEGE moves from routes::datasets (private) to
routes::auth (pub) so routes::traits can reuse it instead of
duplicating the string.
tests/traits.rs and its Python mirror cover both categories (including
both ways of addressing a Publish dataset), the 401/403/404 cases, and
the non-numeric-trait-against-Publish-dataset edge case, run against a
real gn-auth instance. The Publish branch's deny path itself is only
exercised via gn-auth's "unlinked dataset" fallback (already proven for
ProbeSet) -- no non-public PublishFreeze exists in the database this
was tested against.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
List endpoints for the two trait domains, each the listing equivalent
of an existing singular endpoint:
- GET /groups/{group}/traits: every published (phenotype) trait in a
group (db::get_group_traits), same fields/join as
/groups/{group}/traits/{trait_id}, no token required -- matching its
singular sibling, which isn't gated either. A large group (BXD has
15000+ phenotype traits) can take several seconds and tens of
megabytes to return in full; no pagination yet, matching
GeneNetwork2's own wqflask.api.router.fetch_traits, which has no
default limit either.
- GET /datasets/{dataset}/probesets: every expression (ProbeSet) trait
in a dataset (db::get_dataset_probesets), same fields/join as
/datasets/{dataset}/probesets/{probeset}. Gated the same way as that
singular endpoint -- a valid token is required unconditionally, and
a non-public dataset is checked against gn-auth once for the whole
dataset via a new auth::authorise_dataset, rather than once per
probeset in the list. Leaving this listing ungated would have let a
caller read out a whole non-public dataset's trait data in one
request, bypassing the gate just added to the singular endpoint.
auth::authorise_dataset wraps authorise_trait with an empty trait
name: gn-auth resolves a trait's resource purely from its dataset
name, so the trait half of a "Dataset::Trait" string never affects
which privileges come back. Verified directly against the live
instance both for an unlinked non-public dataset (403, deny) and
folded into dataset_probesets's own test for the public/embargo-aged
bypass path.
tests/group_traits.rs, tests/dataset_probesets.rs and their Python
mirrors cover both, run against a real gn-auth instance for the gated
one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Same treatment as dataset_probeset_info: require a valid AuthToken
unconditionally, and for a dataset that isn't publicly readable, check
group:resource:view-resource via authorise_trait -- denied is a 403,
gn-auth unreachable is a 502.
Unify DatasetProbesetLookup/DatasetProbesetInfoLookup back into one
generic enum (with the new Forbidden variant) now that both
dataset_probeset_info and dataset_probeset_values need it, rather than
keeping two near-identical enums.
tests/dataset_probeset_values.rs and its Python mirror gain the same
401/403 cases as dataset_probeset_info's, run against a real gn-auth
instance; the 403 case reuses UTHSC_BXD_Cochlea_0225, confirmed to
have both ProbeSet metadata and ProbeSetData values rows so the test
exercises the authorization check rather than ProbesetNotFound.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Require a valid gn-auth bearer token on this endpoint unconditionally.
For a dataset that isn't publicly readable, check the
group:resource:view-resource privilege via a new auth::authorise_trait,
the "Dataset::Trait"-keyed equivalent of authorise (POST
/auth/data/authorisation instead of /auth/resource/authorisation) --
gn-auth resolves the resource itself from the dataset name, so no
separate resource-id lookup is needed here. A trait never linked to a
gn-auth resource (gn-auth answers 404) is treated as "no privileges",
not a request failure. Denied requests get a 403; a failure to
reach/parse gn-auth's answer gets a 502, distinct from both that and
this service's own 500s.
is_publicly_readable now also takes the dataset's CreateTime and treats
anything older than 5 years as public regardless of its
public/confidentiality flags -- a confidentiality embargo isn't meant
to last forever. No admin/sysadmin bypass is implemented.
Register a "bearer_token" OpenAPI security scheme (src/lib.rs) so the
new security(...) annotation on this endpoint's #[utoipa::path] has
something to point at.
tests/dataset_probeset_info.rs and its Python mirror gain cases for the
missing-token (401) and denied-but-existing (403) outcomes, run against
a real gn-auth instance set up as described in docs/gn-auth.md; the
403 case uses UTHSC_BXD_Cochlea_0225, a real non-public, non-embargoed
dataset never linked into gn-auth. tests/common gains get_with_token
and auth_server_available/auth_token helpers for this.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* docs/gn-auth.md: Document the GN2 public>0 AND confidentiality<1
threshold (the test for whether data needs a gn-auth authorization
call at all, versus just a valid token), the now-implemented
src/auth.rs mechanism, and the four gn-auth bugs found and fixed while
building it.
* RELEASE_NOTES.md: Likewise.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* docs/gn-auth.md: New file. Explains how a client (genenetwork4)
should authenticate callers against gn-auth: tokens are RS256 JWTs,
verified locally via the JWKS at GET /auth/public-jwks (mirroring
genenetwork3's gn3/oauth2/jwks.py, with two improvements -- cache and
periodically refresh the key set instead of refetching per request,
and check exp/nbf/iss, not just the signature); resource-scoped
authorization checks (e.g. "can this token's user do X on InbredSet
Y") via /auth/resource/populations/resource-id/... and
/auth/resource/authorisation, mirroring genenetwork3's
gn3/api/case_attributes.py:required_access. Documents why
POST /auth/introspect should not be used yet (unconfigured client
allow-list, no client ever created with the client_type it requires,
and JWT tokens are never persisted for it to find).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* docs/genenetwork_datatypes.md (ProbeSetXRef): New section, the
expression (mRNA) counterpart to the PublishXRef section above:
ProbeSetXRef's own fields, its relationships to ProbeSetFreeze/
ProbeFreeze/InbredSet/Tissue, ProbeSet, and ProbeSetData/ProbeSetSE/
Strain, and the lack of a declared uniqueness constraint on
(ProbeSetFreezeId, ProbeSetId).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* docs/genenetwork_datatypes.md (PublishXRef): New section, covering
PublishXRef's own fields and its relationships to Phenotype,
Publication, PublishData/PublishSE/NStrain and Strain, based on
topics/systems/mariadb/precompute-publishdata.gmi.
* docs/SKILL.md (mysql crate): Add notes on aliasing joined columns to
match FromRow's UpperCamelCase conversion instead of reaching for
#[mysql(rename = "...")], and on mysql::Pool::new's expensive default
PoolConstraints (eagerly opens 10 connections) making it unsafe to
call per request.
(Testing): Add timing DB-backed test requests through a shared helper,
since that's what actually caught the Pool::new issue above.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* docs/SKILL.md (Guix packaging): Replace the generic "pin crates
down or bump rust" note with the concrete sequence used to fix this
in genenetwork4: one-crate-at-a-time --precise pins (and why
re-running generate-lockfile undoes them), the mysql_common
let-chains/is_multiple_of case where no patch version within a minor
line helps and the dependent itself needs downgrading, and querying
the sparse crates.io index directly to compare MSRVs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* docs/SKILL.md: New file. Covers the mysql crate's feature/FromRow
quirks, the database-name-vs-code-name convention, the ApiError/
request-id error shape, utoipa OpenAPI setup, the two-suite test
pattern, Guix packaging traps, and commit style, as practiced building
this repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* docs/genenetwork_datatypes.md (Species)[Fields]: Pad table columns
with spaces so pipe characters line up in a plain editor.
(InbredSet): New section. Field semantics (sourced from the live
schema and genenetwork2/gn2/base/data_set/datasetgroup.py's
MappingMethodId mapping), relationships to PublishFreeze/GenoFreeze/
ProbeFreeze/StrainXRef, known quirks, and a survey of existing
implementations across GeneNetwork2, GeneNetwork3, gn-guile and this
repo. States "Group" as the recommended alias for InbredSet, backed by
GeneNetwork2's /api/v_pre1/groups and /api/v_pre1/group/<name> routes,
the DatasetGroup class docstring, the group_name variable convention
used throughout GeneNetwork2 and GeneNetwork3, and the "Groups" heading
in gn-hacking-documentation.gmi; notes "RISet" as a secondary, more
genetics-specific alias that is a slight misnomer for InbredSet rows
whose GeneticType is not an actual recombinant inbred set.
|
|
* docs/genenetwork_datatypes.md: New file. Species table field
semantics (sourced from factory/docs/gn-gemtext-threads/topics/
gn-uploader/adding-species.gmi and this repo's own
docs/mariadb_db-webqtl.md schema dump), its relationships to
InbredSet, Strain, Chr_Length and GeneIDXRef, its role in GN's
authorization and RDF classification scheme, known quirks
(mouse-centric legacy genotype columns, undocumented FamilyOrderId/
OrderId, no UI for adding a species), and a survey of existing
Species-related implementations across GeneNetwork2, GeneNetwork3,
gn-guile and this repo.
|
|
* docs/mariadb_db-webqtl.md: New file. CREATE TABLE statement for
each of the 112 tables in db_webqtl, dumped with mysqldump --no-data
and grouped under a linked table of contents.
|