about summary refs log tree commit diff
path: root/docs
AgeCommit message (Collapse)Author
12 daysrqtl: refuse a phenotype whose sample count is not the cross'sPjotr
Asked to map a 9-strain trait, the wrapper scanned it against the 6-individual fixture cross and returned six markers with LOD scores and `state: complete` -- a plausible answer belonging to nobody. R recycled and truncated the values silently, and nothing downstream could tell. GeneNetwork3's copy of this script has the same hole. The cross-file format carries no per-sample names, so the count is the only check available; order remains the caller's responsibility (and the reason values and genotype file must come from the same panel). With the check, the same request is a 502 rqtl_failed quoting both counts, and the correctly-shaped request still returns the documented LODs. * scripts/rqtl_wrapper.R: compare the pheno file's rows with nind() of the cross, and stop. * README.md (/compute/rqtl): the second fix relative to gn3, and what stays the caller's job. * docs/www/ai-instruction.md (4a): values and genotypes must describe the same individuals, one for one, in the same order. * docs/www/operator-checklist.md (Compute): what genotype files actually exist on this box (test fixtures, 198 BXD individuals) and why a database trait, with its own strain subset, cannot be mapped against them.
12 daysvalues: suggest R/qtl and GEMMA in the envelope's `actions`Pjotr
Fetching one trait's value per strain is usually the step before mapping that trait, and the API can do the mapping. Those calls are POSTs with a body, so they cannot go in `next` -- a `next` link is a GET that returns 200, which is what makes `next` safe to follow blindly and what tests/envelope.rs relies on. Hence a fifth envelope member, `actions`, holding method + href + description + body, and omitted entirely when there is nothing to suggest (most responses keep the four-member shape). The three per-trait values endpoints now carry a `POST /compute/rqtl` and a `POST /compute/gemma` suggestion with the values already in the body, in the response's own sample order. Genotype/SNP paths stay bracketed placeholders: they must exist on the server, which the server cannot know on the caller's behalf. The instruction page says, in the same breath, never to run R/qtl or GEMMA locally -- a local scan would use genotypes that are not the panel's and would disagree with GeneNetwork's own results -- and that results are cached, so repeat calls are cheap (measured: ~2 s cached against ~20 s cold on the BXD fixtures). Also fixes a flake the new test exposed rather than caused: config::tests read and write process-global env vars and Config::load() reads all of them, so the RAVANAN_* tests could fail search_engine_keys_follow_env_over_default_layering on an unrelated assertion. One module-level lock serialises them. * src/response.rs (SuggestedCall, Envelope::actions/with_action/ with_actions): the new member, omitted when empty; unit tests for both. * src/suggest.rs: new. Builds the two suggestions for either value type (db::TraitValue, db::ProbeSetValue), with its own unit tests. * src/routes/{traits,groups,datasets}.rs: the three values handlers. * tests/compute_suggestions.rs: new. Suggestions present with values in order, on all three routes; absent elsewhere; `next` still GET-only and still returns 200. * src/config.rs (tests): env_guard, taken by the 11 config tests. * README.md (Success responses): the `actions` member, with an example. * docs/www/ai-instruction.md (4a): never compute locally, why, caching, and how to use the suggested bodies. * docs/www/operator-checklist.md: the caching numbers.
12 dayscompute: proxy /compute/* for any valid tokenPjotr
R/qtl and GEMMA are part of the service now, so the blanket 403 goes: the handlers already require an AuthToken, which is the whole gate ("token_only", not resource-checked). What that buys and costs is now written down rather than implied -- any token holder can name server-side paths for R and gemma to read, and the public-data-only restriction on traits/datasets does not apply here at all. The box also has to be able to run them: PATH must carry Rscript, gemma-wrapper and gemma, *and* R_LIBS_SITE must point at the profile's site-library or scripts/rqtl_wrapper.R dies on `there is no package called 'optparse'` -- which reads like a token or proxy fault and is neither. Verified through nginx with a token: POST /gn4/compute/rqtl 200 in 0.4s with the fixture's marker rows, POST /gn4/compute/gemma 200 in 2.1s with 7 320 marker rows (identical to the documented run's first row), GET /gn4/compute/gemma/status/<id> ending in "GN4: computation complete", /compute/correlation 200; all four 401 without a token. * etc/nginx-vesuvio.pu.ac.ke.conf: drop the /compute 403 locations, keep them commented as the way back; timeouts/body-size comments now apply. * docs/www/ai-instruction.md: new §4a, "Running QTL mapping (compute)" -- shapes, server-side paths, ?id= + status polling, what to do with a 504. * docs/www/operator-checklist.md: compute moves from "must be off" to "proxied, know what it grants", plus what the box needs (PATH, R_LIBS_SITE, ravanan) and the measured results. * docs/nginx.md: same, in the deployment and checking sections.
12 daysdocs/www: point the assistant at the repository, not the forge UIPjotr
The instruction page is now also the way to get the rest of the docs: clone the repo and read docs/. Measured, so the numbers are real -- a --depth 1 --sparse clone of docs is ~2.5 MB in a couple of seconds; the plain --depth 1 clone is 17 MB, almost all of it source tree the reader does not need. Why clone rather than fetch: the forge's tree pages sit behind Anubis (a browser proof-of-work challenge) and its raw endpoint answers 403, so a plain HTTPS fetch of a Markdown file returns an HTML challenge page. The git smart-HTTP endpoints pass, so clone works -- which is what makes this instruction usable at all. Also says how to tell whether the docs match the running build (`git log -1 --format=%h` against the `git` field of /gn4/version). * docs/www/ai-instruction.md: "Get this document, and the rest of the docs, by cloning", ahead of the numbered sections.
12 daysdocs: search is Xapian, and what that means for a queryPjotr
The AI instructions described the field prefixes but not the semantics, so a reader would reasonably assume AND between words, working quoted phrases and wildcard prefixes -- none of which is true here. Measuring against the production index: juxtaposition is OR (`alcohol` 1 200 vs `alcohol AND dependence` 45), free text is stemmed and case-insensitive, quoted phrases and NEAR/ADJ match *nothing* because the stemmed terms carry no positions (a hyphenated word is a phrase, so `insulin-like` also matches nothing), and wildcards are not enabled (`alcohol*` is just `alcohol`). * docs/www/ai-instruction.md (3. Search): rewritten around the Xapian QueryParser semantics and a table of exact / stemmed / numeric-range fields. * docs/search-engines.md: the same, for whoever maintains the engine, plus the gn3-forward-vs-native difference in what counts as a field.
12 daysdocs/www: say who issues tokens, and what the assistant should ask forPjotr
The page said a human issues tokens and to "stop and ask", without saying whose door to knock on -- and the operator commands belong in the operator file anyway. State it as the GeneNetwork team, with wording the assistant can relay to its user: what a token is for, how long the default lasts, and to pass it out of band rather than through a public issue or a shared chat. Same wording for renewal, since a 401 is the one failure the assistant cannot fix itself. * docs/www/ai-instruction.md (1. Get a token): "If you do not have one" and "If it stops working", the latter quoting request_id and retrying exactly once; the summary line and the 401 row of the error table say GeneNetwork team rather than "the operator". * docs/www/operator-checklist.md (Token issuance): the request path, so the team knows what the ask they receive looks like. * .gitignore: *.swp, after one got swept into this commit.
12 daysdocs/www: rewrite the AI instructions for the live deploymentPjotr
ai-instruction.md was written before anything was deployed: it took a placeholder base URL, did not say where the API is mounted, and buried the operator's deployment notes in an appendix labelled "not for the AI reader" -- awkward to hand to an assistant, and confusing when it is. So: the AI-facing page now names the deployment (https://vesuvio.pu.ac.ke/gn4) and is self-contained, and the operator material moves to operator-checklist.md. Everything in the page was exercised against the running deployment. Content changes, beyond the move: * `next` hrefs arrive already prefixed (`/gn4/groups/BXD/strains`), so they resolve against the origin -- joining them to the current URL's directory double-prefixes them. Said plainly, since it is the one thing the prefix breaks. * says what cannot work and should not be retried: get-test-token is a 404, /compute/* is a 403 from the front end, and neither is a permissions problem to work around. * `/publications/{pubmed_id}/traits` wants the PubMed id, not the internal `publication_id` in the same trait record, which 404s. * states the current rate-limit position (none configured) instead of implying a 429 policy that does not exist yet. * docs/www/ai-instruction.md: rewritten, paste-ready. * docs/www/operator-checklist.md: new file, the former appendix, updated with what is now enforced at the proxy and a record of the verified request matrix. * README.md, docs/nginx.md, etc/nginx-vesuvio.pu.ac.ke.conf: point at the operator checklist by its new name.
12 daysdocs: SEARCH_ENGINE=native in the nginx deployment's environmentPjotr
Without it /search forwards to GeneNetwork3 at GN3_URL (:5003) and answers 502 backend_unreachable when nothing is listening there -- a correct error, with the error body's links rewritten, but easy to misread as the proxy being wrong. The deployment has an index, so native is what it wants. * docs/nginx.md (The server behind nginx): add the variable and say what the default does. * etc/nginx-vesuvio.pu.ac.ke.conf: same, in the start-up comment.
12 daysdocs: a certbot-edited default site silently wins over this blockPjotr
Reproduced from a mirror of the deployed /etc/nginx: with sites-enabled/default still present, nginx logs conflicting server name "vesuvio.pu.ac.ke" on 0.0.0.0:443, ignored and answers 443 with the default site -- stock welcome page at /, nginx's own 404 for /gn4/..., HTTP/1.1 despite `http2 on` in this block. sites-enabled/* is globbed alphabetically and the first block whose server_name matches wins, so `default` (edited by certbot to serve this hostname) shadows the gn4 block. Removing it gives the expected 302 / 301 / 200 / 404 / 403 matrix. * docs/nginx.md (nginx): say to install the snippet instead of the distribution default, and what the shadowing looks like from outside.
12 daysetc: the host is vesuvio.pu.ac.ke, not .ac.ukPjotr
The certificate on :443 carries subjectAltName DNS:vesuvio.pu.ac.ke, and vesuvio.pu.ac.uk does not resolve at all, so the config as committed could not have matched a request for the real name: nginx falls through to the default server, which is why /gn4/... answered 404 from nginx rather than a 502 from genenetwork4. * etc/nginx-vesuvio.pu.ac.uk.conf -> etc/nginx-vesuvio.pu.ac.ke.conf: server_name and the certificate paths. * docs/nginx.md: the same rename, and a "Checking it" section that starts from what actually gets in the way on the box -- TLS validates the URL hostname rather than a Host: header, so on-box checks need --resolve against the name on the certificate, and a mismatch there and in server_name look identical (default server answers).
12 daysetc: drop-in nginx config for vesuvio.pu.ac.uk/gn4Pjotr
docs/nginx.md explained the pieces; this is the assembled file for the deployment actually being set up, so it can be reviewed and installed without stitching anything together. It lives in etc/ rather than docs/, being deployable configuration rather than prose. It refuses /compute/*, which the README-based template did not: those endpoints take server-side file paths and run gemma/ravanan/R on them, so any token holder able to reach them can read the box -- the operator checklist in docs/www/ai-instruction.md already said to keep them on the internal listener, and this is that rule where it belongs. * etc/nginx-vesuvio.pu.ac.uk.conf: new file. TLS, http2, log format carrying x-request-id, / and /gn4 redirects, proxy_pass with the prefix-stripping trailing slash, raised timeouts, sub_filter rewriting of href/next/spec_url/type, and 403/404 refusals for /compute/ and /get-test-token. * docs/nginx.md: point at it, and say why /compute/ is refused.
13 dayssmall editpjotr
13 daysai-instruction: issue a token per user without gn-authPjotr
The page told the operator to mint with POST /get-test-token, which is gn-auth's dev password grant and does not exist in the deployment being set up. Issuing a second user's token also had no good answer: the stub minted exactly one token per process, so a new user meant starting a throwaway instance, grabbing --token-file and killing it. So add --mint-only: mint for --subject, print, write --token-file and exit without listening. Verification only depends on the key (committed, so shared by every instance) and on iss, which now falls back to AUTH_SERVER_URL -- a token minted this way verifies against the long-running stub untouched. * scripts/gn-auth-stub.py (main): --mint-only; --issuer defaults to AUTH_SERVER_URL before the port-derived default. Docstring says how a second user's token is issued. * docs/www/ai-instruction.md (1. Get a token): the operator's command is the stub, not /get-test-token; the token lasts whatever --expires-in said rather than an hour, and there is no early revocation. (4. What is public): minting is operator-side. Appendix: "Token issuance" rewritten around the stub -- public-data-only, expiry as the only revocation, sub as attribution, one token per user -- with gn-auth kept as what replaces it later; the stub is no longer listed as a dev leftover to delete, but as the auth server that must never be routed publicly. * README.md (Tests): mention --mint-only next to the stub's other flags.
13 daysdocs: publish genenetwork4 under a path prefix with nginxPjotr
The deployment being set up is https://vesuvio.pu.ac.uk/gn4 proxying to a loopback port, with no gn-auth yet. Two things there are not obvious and neither is discoverable from the README's /get-test-token section: every link the API emits is root-relative, so under /gn4 they resolve outside the mount point; and /compute/* runs for minutes in a single request, so nginx's one-minute default read timeout cuts it off mid-computation. The link rewriting is the delicate part -- the server's serde output is compact ("href":"/"), not the pretty-printed shape the README examples show, so a filter written from those examples silently does nothing. Both spellings of all four affected keys (href, next, spec_url, type) are in the config, and the envelope's "next":[ array is deliberately left alone. * docs/nginx.md: new file. Server environment, the nginx block, what the prefix does to links and the three ways to handle it, stub-minted tokens instead of gn-auth, and how to check the result. * README.md (Behind nginx or in a container): point at it.
13 daysdocs: how an AI client uses the public API, and what must not leakPjotr
docs/www/ai-instruction.md is the AI-facing page: you are handed a JWT (the operator mints it on the server's loopback interface, so there is no token endpoint on the public host and no flow for the model to follow), send it as a bearer token, and read the envelope. It covers discovery (/openapi.json, /version), the readable endpoints, search's grammar and paging cost, the public-data rule (public > 0 && confidentiality < 1, or older than five years), error classes and what to do with each, and a worked example. The operator appendix records the deployment findings from this session: * /get-test-token is now the minting route and only a loopback peer check plus proxy-header refusal keeps it off the internet -- it must not be proxied and the app must stay bound to 127.0.0.1; * POST /compute/* takes server-side file paths and is token_only, so it is an arbitrary-read/execution surface and stays internal; * the public rule is enforced per dataset for values, while metadata endpoints are token_only by design and do reveal non-public dataset records -- a decision to make, not an accident; * the index must be built with the public filter (GN4's indexer does; gn3's index-genenetwork has no WHERE clause). Verified on the current full index: the confidentiality=1 HCPPublish and the public=0 BR_U_0303_M dataset both return zero hits; * error detail can carry the database address, and tokens minted this way all carry gn-auth's development user as sub, so the audit trail is "someone with a token" until client_credentials exists.
13 dayssearch: lowercase boolean prefix valuesPjotr
The index stores every boolean term in lowercase and holds no mixed-case variants at all: XDSbxdpublish (15 060 docs), Awilliams (10 958), XSmouse, XCx, XYbdnf, XImrna all exist, while XDSBXDPublish, AWilliams, XSMouse, XCX, XYBDNF and XIMRNA report "not in database" on both the dev index and a full production one. Matching the value exactly therefore rejected the way a caller naturally writes it -- dataset:BXDPublish returned 0 hits against an index holding 15 060 BXDPublish traits, author:Williams 0 against 10 958 williams authors. Lowercase the value for every boolean prefix, generalising the group-only rule gn3 already has (search.py:204-209). No graded parity shape changes: they all use lowercase values already. Pinned per prefix in boolean_prefixes_parse_to_exact_index_terms, so a registration that regresses to an exact add_boolean_prefix fails there instead of silently returning zero hits.
2026-11-05docs: what the slurm backend costs, and why job records must stayPjotr
Measured the same request (BXD fixtures, three chromosomes) through /compute/gemma, ravanan's single-machine backend and ravanan on Slurm, cold and warm. The Slurm backend is dispatch-bound at this size -- one scheduler tick per step, with each job itself running in under a second -- so cold and warm are identical there, and the README's 16.7s figure belongs to the single-machine backend. Also document the Slurm-side requirement behind that measurement: a completed job has to stay queryable for as long as the workflow runs. Once slurmctld purges the record ravanan falls back to slurmdbd, and on a cluster without accounting the workflow aborts on a 404 long after the job succeeded.
2026-10-24cwl: Document how a ravanan+Guix+Slurm request actually travels.Pjotr
cwl/README.md explains the whole path (API -> ccwl compile -> ravanan -> slurmrestd -> slurmd), where Guix sits (front end only: nodes just need the same /gnu/store mounted, plus bash, a writable ravanan store for data/logs, and scratch for the job's working directory), the two caches (ravanan's work reuse, checked before submitting; gemma-wrapper's per-chromosome --cache-dir, inside the job) and why the latter is what actually saves work for this endpoint. It also answers "can the nodes rebuild the tools instead of sharing a store": yes in principle, since Guix is deterministic and a node building the same derivations would materialise identical store paths, but ravanan submits a store path and Slurm does not substitute automatically -- so it needs a realise-then-run wrapper (ravanan's slurm backend already holds the script's derivation), a per-workflow realise job, or guix publish plus a node-side daemon, with the costs and channel-pinning caveats listed. The timing table from the test cluster (cached: identical; fresh: waves vs synchronous steps) is included so expectations are explicit. docs/ravanan-slurm.md and README.md link to it.
2026-10-24gemma: Make ravanan's batch system configurable (optional Slurm).Pjotr
POST /compute/gemma/ravanan used to run every CWL step locally, because compute_ravanan never passed ravanan a batch system. Add a RavananBatch setting: - RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing closed on anything else, exactly like GN4_DB_MODE; - slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint, --slurm-jwt (a *file*, which is what ravanan reads) and, when set, --slurm-partition/--slurm-nice; a slurm selection without an endpoint or a JWT file refuses to start rather than 502-ing every request; - --scratch defaults to ravanan_cache_dir for slurm, so the store and the job scratch space live on the same shared filesystem (with slurm that directory has to be visible to the compute nodes); - the response's ravanan note records which backend ran. The switch building is RavananBatch::ravanan_args so it is unit testable; compute_ravanan takes the backend and the scratch directory, and the route passes them from config. docs/ravanan-slurm.md documents the keys, the shared-directory requirement, and the slurm-side auth wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in slurmrestd's environment) that makes REST-submitted jobs actually run.
2026-10-13docs: align and width-cap markdown tablesPjotr
Realigns every pipe-table column to a uniform width, and trims cell text so each table stays under 100 chars per row, moving one over-long cell's detail (InbredSet.MappingMethodId) into a footnote paragraph below its table instead of dropping it. The gn-auth.md endpoints table is left over 100 chars, with a note explaining why: its longest path is 66 chars by itself, leaving no room for the other two columns under that cap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-13auth: add gnauth compile-time tags (secured/token_only/disabled)Pjotr
Adds a gnauth-macros proc-macro crate and a private-field Authorized capability type in src/auth.rs, so #[gnauth(secured)] is a real, compile-time-checked guarantee that a handler performs the gn-auth resource-authorisation round trip before sharing trait/probeset values, rather than just a comment. #[gnauth(token_only)] marks handlers that require a token but skip the round trip, and #[gnauth(disabled)] marks the two handlers that need no token at all. Also fixes groups.rs: group_trait_values, group_trait_info and group_traits previously performed no gn-auth round trip at all for published-trait data; they now resolve the group's PublishFreeze row and call the same auth::check used elsewhere, denying by default when no such row exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-08* src/routes/dev.rs: POST /get-test-token answers local callers only, not ↵Pjotr
tied to dev mode. The endpoint no longer depends on GENENETWORK4_DEV_MODE or on the bind address. It decides per request whether to answer. * src/routes/dev.rs (get_test_token): answers a caller only when the TCP peer (axum ConnectInfo) is in 127.0.x.x (or ::1, or an IPv4-mapped 127.x) and the request carries no X-Forwarded-For, Forwarded or X-Real-IP header (a reverse proxy on the same host would otherwise make every client look local). Anything else, including an unknown peer, is a 403. Local is never established from a header. * src/main.rs: serve with into_make_service_with_connect_info, which provides the peer address. * src/config.rs: remove dev_mode, GENENETWORK4_DEV_MODE and dev_token_enabled(). Add GENENETWORK4_TEST_TOKEN (on by default); when it is not a clearly affirmative value (true/1/yes/on) the route is not registered: a 404, absent from /openapi.json. It fails closed, so a typo ("flase") turns the endpoint off. * src/lib.rs: register the route unless that switch is off. * tests: tests/get_test_token.rs rewritten (remote, unknown and proxied peers refused; 127.x and ::1 answered; always documented by default); tests/get_test_token_off.rs for the switch (own process: it sets an environment variable); common::post_from supplies a peer address to the in-process requests. Python: test_get_test_token.py rewritten, TOKEN_ENV is now empty (no setting needed). * README.md, docs/gn-auth.md, docs/SKILL.md, RELEASE_NOTES.md: document the check, the switch, and "Behind nginx or in a container" (a bridge container sees the gateway as the peer; a host-network container behind nginx sees 127.0.0.1, so block the path in nginx, set X-Forwarded-For, or switch the endpoint off). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/db.rs, src/routes: add mapping_method, tissue, /mapping-methods, ↵Pjotr
/accessions. Answers four more questions of aisearch's full_benchmark.csv from tables the API did not expose. * src/db.rs (Group, GroupDetail), src/routes/groups.rs: groups gain `mapping_method`, the name of the MappingMethod row their mapping_method_id points to (qtlreaper, Rqtl, ...), or null. * src/db.rs (Dataset): datasets gain `tissue`, the Tissue their ProbeFreeze points to (e.g. "Brain mRNA"). * src/routes/mapping_methods.rs: GET /mapping-methods lists the MappingMethod table, including tools no group uses (PLINK, FastMap). * src/routes/accessions.rs: GET /accessions/{accession} resolves a GeneNetwork accession number (GN602, gn602 or 602) via InfoFiles to the dataset's name, title, category (publish/probeset) and group. * docs/genenetwork_datatypes.md: MappingMethodId is a key into the MappingMethod table (the authoritative reading); GN2's code-based reading is noted as superseded. * tests/benchmark.rs, test/data/benchmark_search.json: add the 60 new full-benchmark rows that pass (full rows 0..114; search rows run against recorded production gn3, the rest need the database), and full rows 33 (mapping tools), 34 (Rqtl for outbred sets), 35 (GN602) and 44 (only GTEx v8 has TPM datasets). The untested rows are listed in the file's header comment with the reason for each. * tests, test/python: tests for the new fields and endpoints; both endpoints added to the OpenAPI and next-link crawl tests. * README.md, RELEASE_NOTES.md: document the new fields and endpoints. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/response.rs: wrap every success response in one envelope.Pjotr
Breaking: every success body is now {"data", "meta": {"state", "took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only `/` (plain text) and `/openapi.json` are unwrapped. * src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers take Ctx first and build the response with ctx.envelope(data), so timing and the request id are never filled in by hand. `state` is `complete` or `partial`; notes are server-authored only; `next` are concrete relative hrefs (path_segment percent-encodes names). * src/routes/*.rs, src/lib.rs (version): every handler returns Envelope<..> with next links; /search is `partial` when the locus lookup fails or a sorted 1000-hit window may be truncated, and /publications/{pubmed_id}/traits when datasets were left out for authorization. * src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member. * src/auth.rs (AuthToken): the 401 now carries request_id and took_ms (it never had a request_id); Ctx stores RequestInfo in the request extensions for it. * tests: common::data / envelope_data parse every success body through the envelope; tests/envelope.rs and test/python/test_envelope.py check the shape, errors, and that every `next` link resolves; tests/envelope_partial.rs checks the `partial` search without a database; tests/openapi.rs fails if a JSON success response is not an Envelope_* schema. * README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document the envelope and took_ms on errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08docs/search-engines.md: the indexer is gn3's scripts/index-genenetworkAlexander_Kabui
-- name the repo so the command is findable from gn4.
2026-10-08docs/search-engines.md: give the indexer command its scriptAlexander_Kabui
prefix (scripts/index-genenetwork, verified against the click CLI); plain 'planned response enrichment' instead of naming a proposal; state the behavior-preserving default up front.
2026-10-08docs/search-engines.md: how to select and run a search engine (theAlexander_Kabui
gn3 proxy by default, the native xapian engine on SEARCH_ENGINE=native + XAPIAN_DB_PATH); README.md (search): point to it. House style: scope line, takeaway headings, code pointers, known quirks (locus null reason, sort tail difference, no runtime indicator).
2026-10-05doc: Add auth/testing insights to docs/SKILL.mdPjotr Prins
Captures the reusable practices from this session's token-gating work, for reuse on a similar Rust/Axum service: - utoipa: a document-wide default security(...) on #[derive(OpenApi)] instead of repeating it per operation, with per-operation empty security() overrides for the exceptions; a shared #[derive(utoipa::IntoResponses)] type for a response shape (e.g. 401) repeated across many handlers, referenced bare in responses(...). - Authentication/authorization: an opt-in-per-handler extractor (AuthToken) over blanket middleware; keeping "is this token valid" and "is this specific resource public" as independent checks; resolving a (container, item)-addressed resource's container category before ever looking up item, rather than guessing item across categories; a dev-only token-issuing endpoint needing at least two independent config conditions (not runtime-detected), registered conditionally rather than registered-and-rejected, with credentials defaulting to the target service's own published dev fixtures. - Testing: a test suite fetching its own token from that dev endpoint (in-process for an in-process Rust suite, via per-test-class subprocess env for a spawned-binary Python suite) instead of requiring a human to provision one; the concurrency hazard of mutating process-wide env vars across #[tokio::test] functions in one file, and why a spawned-subprocess-per-class suite doesn't have it; Rust edition 2024's unsafe env::set_var/remove_var. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-05Rename /dev/token to /get-test-token; require a token everywherePjotr Prins
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs, TestToken replacing DevToken), matching tests/get_test_token.rs and test/python/test_get_test_token.py (renamed from tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and the GN_AUTH_DEV_* config keys keep their names -- they're about the dev-token feature generically, not tied to this one route's path. Every endpoint now requires a valid token, not just the dataset_probeset*/traits* family: species(_by_name/_datasets), groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all gain an AuthToken parameter. /, /version, /openapi.json and /get-test-token itself are the only exceptions -- a health check, a version check, the spec describing all this, and the endpoint that hands out a token in the first place (which would be circular if it needed one). Rather than repeating security(("bearer_token" = [])) on every #[utoipa::path], ApiDoc now declares it once as the document's default security(...), and the three exceptions besides /get-test-token (/, /version) override it with their own empty security() -- standard OpenAPI semantics (an operation's own security, even empty, overrides the document default; an absent one inherits it). error::Unauthorized, a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the shared 401 every gated handler now references bare in its own responses(...) list instead of writing out the same (status = 401, description = "...", body = ApiError) tuple by hand. Verified the resolved security per operation in a live /openapi.json. tests/common::auth_token() (Rust) and the new test/python/auth_helpers.auth_token() (Python, deduplicating what four files previously each defined locally) now try this server's own POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN -- every previously-ungated test file gains a token check and switches to get_with_token; the whole suite, both languages, now runs with zero manually-provisioned tokens given GENENETWORK4_DEV_MODE=true GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains EXTRA_ENV so each Python test class can start its own server with dev mode on. Verified both configured (auto-fetch) and unconfigured (GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g. the Guix build sandbox) paths. README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for the rename and the new blanket token requirement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-05Add POST /dev/token, a dev-only gn-auth token issuerPjotr Prins
Calls gn-auth's own password grant (POST /auth/token) for gn-auth's well-known development user and returns the result, so local testing no longer needs a manual curl/gn-auth round trip of its own. Only ever registered -- not just rejected at request time, genuinely absent from the router and from /openapi.json -- when two new Config fields are both set: dev_mode (GENENETWORK4_DEV_MODE) and a loopback bind (GENENETWORK4_BIND, also new). Both are config, not something genenetwork4 infers from its actual network exposure, by design: dev mode alone, or a loopback bind alone, each leave it disabled, and the check (Config::dev_token_enabled) is a plain function so the condition itself has direct unit test coverage rather than only being exercised indirectly through the route. src/main.rs now binds to config.bind instead of a hardcoded "0.0.0.0", via a proper SocketAddr rather than string formatting -- the latter would mishandle an IPv6 bind address like "::1" (needs brackets). GN_AUTH_DEV_CLIENT_ID/_CLIENT_SECRET/_USERNAME/_PASSWORD default to gn-auth's own published dev fixtures (flask init-dev-clients, gn_auth/wsgi.py), so this works unconfigured against an unmodified dev gn-auth instance -- confirmed live, including using the returned token against a real gated endpoint (/traits/BXD/10001), not just checking the token response's shape. tests/dev_token.rs covers all four combinations of the two settings in one sequential test function (mutating the same two env vars from more than one #[tokio::test] in a file would race, since they run concurrently by default) plus the /openapi.json visibility check. test/python/test_dev_token.py mirrors this using separate test classes instead, which is safe here since each spawns its own server subprocess with its own environment (ServerTestCase gains EXTRA_ENV for this). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /traits/{dataset}/... spanning ProbeSet and Publish dataPjotr Prins
A new, additional endpoint family that addresses both trait categories the same way -- (dataset, trait) -- matching GeneNetwork2's own generic wqflask.api.router.trait_sample_data/get_trait_info, rather than the two category-specific shapes genenetwork4 already has (/datasets/{dataset}/probesets... and /groups/{group}/traits...), which are unchanged and keep working exactly as before. - db::resolve_trait_dataset(name) resolves a dataset name against ProbeSetFreeze first, then the new db::PublishDataset (a PublishFreeze row, via the new get_publish_freeze_by_name) -- whichever matches decides the category. This means only dataset names are ever "guessed" across categories, not trait names/ids, which is the much smaller and safer surface: GN2's own equivalent tries ProbeSet.Name and PublishXRef.Id against the same bare trait string and silently falls back, which can return the wrong trait on a same-named collision. get_publish_freeze_by_name accepts a PublishFreeze's own Name/ShortName/Id or its owning group's Name/InbredSetName/FullName/ Id, mirroring GN2's own flexible matching (e.g. "BXDPublish" or "BXD"). - GET /traits/{dataset}: list every trait in a dataset, tagged by category. - GET /traits/{dataset}/{trait}: one trait's metadata. A non-numeric trait against a Publish dataset is a 404 (can't match a PublishXRef.Id), not a 500. - GET /traits/{dataset}/{trait}/values: that trait's per-strain values. TraitValue and ProbeSetValue are already identical in shape, so this unifies with no information loss, unlike the metadata endpoint above. Metadata responses are a tagged union ({"category": "probeset"| "publish", ...}) wrapping ProbeSetInfo/TraitInfo unchanged, rather than a lossy common-fields schema -- the two share little beyond mean/locus/lrs/additive. utoipa generates this as a oneOf with a category discriminator, verified against /openapi.json. Gated the same way as /datasets/{dataset}/probesets..., extended to Publish data for the first time: /groups/{group}/traits itself is still not gated, but reaching the same data through /traits/{group} now requires a token and checks PublishFreeze.public/confidentiality. No gn-auth-side change was needed -- POST /auth/data/authorisation already dispatches *Publish-suffixed dataset names to its phenotype resource resolver (gn_auth/auth/authorisation/data/phenotypes.py), the exact mechanism this integration already relies on for mRNA data. auth::VIEW_RESOURCE_PRIVILEGE moves from routes::datasets (private) to routes::auth (pub) so routes::traits can reuse it instead of duplicating the string. tests/traits.rs and its Python mirror cover both categories (including both ways of addressing a Publish dataset), the 401/403/404 cases, and the non-numeric-trait-against-Publish-dataset edge case, run against a real gn-auth instance. The Publish branch's deny path itself is only exercised via gn-auth's "unlinked dataset" fallback (already proven for ProbeSet) -- no non-public PublishFreeze exists in the database this was tested against. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /groups/{group}/traits and GET /datasets/{dataset}/probesetsPjotr Prins
List endpoints for the two trait domains, each the listing equivalent of an existing singular endpoint: - GET /groups/{group}/traits: every published (phenotype) trait in a group (db::get_group_traits), same fields/join as /groups/{group}/traits/{trait_id}, no token required -- matching its singular sibling, which isn't gated either. A large group (BXD has 15000+ phenotype traits) can take several seconds and tens of megabytes to return in full; no pagination yet, matching GeneNetwork2's own wqflask.api.router.fetch_traits, which has no default limit either. - GET /datasets/{dataset}/probesets: every expression (ProbeSet) trait in a dataset (db::get_dataset_probesets), same fields/join as /datasets/{dataset}/probesets/{probeset}. Gated the same way as that singular endpoint -- a valid token is required unconditionally, and a non-public dataset is checked against gn-auth once for the whole dataset via a new auth::authorise_dataset, rather than once per probeset in the list. Leaving this listing ungated would have let a caller read out a whole non-public dataset's trait data in one request, bypassing the gate just added to the singular endpoint. auth::authorise_dataset wraps authorise_trait with an empty trait name: gn-auth resolves a trait's resource purely from its dataset name, so the trait half of a "Dataset::Trait" string never affects which privileges come back. Verified directly against the live instance both for an unlinked non-public dataset (403, deny) and folded into dataset_probesets's own test for the public/embargo-aged bypass path. tests/group_traits.rs, tests/dataset_probesets.rs and their Python mirrors cover both, run against a real gn-auth instance for the gated one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Gate GET /datasets/{dataset}/probesets/{probeset}/values on gn-authPjotr Prins
Same treatment as dataset_probeset_info: require a valid AuthToken unconditionally, and for a dataset that isn't publicly readable, check group:resource:view-resource via authorise_trait -- denied is a 403, gn-auth unreachable is a 502. Unify DatasetProbesetLookup/DatasetProbesetInfoLookup back into one generic enum (with the new Forbidden variant) now that both dataset_probeset_info and dataset_probeset_values need it, rather than keeping two near-identical enums. tests/dataset_probeset_values.rs and its Python mirror gain the same 401/403 cases as dataset_probeset_info's, run against a real gn-auth instance; the 403 case reuses UTHSC_BXD_Cochlea_0225, confirmed to have both ProbeSet metadata and ProbeSetData values rows so the test exercises the authorization check rather than ProbesetNotFound. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Gate GET /datasets/{dataset}/probesets/{probeset} on gn-authPjotr Prins
Require a valid gn-auth bearer token on this endpoint unconditionally. For a dataset that isn't publicly readable, check the group:resource:view-resource privilege via a new auth::authorise_trait, the "Dataset::Trait"-keyed equivalent of authorise (POST /auth/data/authorisation instead of /auth/resource/authorisation) -- gn-auth resolves the resource itself from the dataset name, so no separate resource-id lookup is needed here. A trait never linked to a gn-auth resource (gn-auth answers 404) is treated as "no privileges", not a request failure. Denied requests get a 403; a failure to reach/parse gn-auth's answer gets a 502, distinct from both that and this service's own 500s. is_publicly_readable now also takes the dataset's CreateTime and treats anything older than 5 years as public regardless of its public/confidentiality flags -- a confidentiality embargo isn't meant to last forever. No admin/sysadmin bypass is implemented. Register a "bearer_token" OpenAPI security scheme (src/lib.rs) so the new security(...) annotation on this endpoint's #[utoipa::path] has something to point at. tests/dataset_probeset_info.rs and its Python mirror gain cases for the missing-token (401) and denied-but-existing (403) outcomes, run against a real gn-auth instance set up as described in docs/gn-auth.md; the 403 case uses UTHSC_BXD_Cochlea_0225, a real non-public, non-embargoed dataset never linked into gn-auth. tests/common gains get_with_token and auth_server_available/auth_token helpers for this. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04doc: Update gn-auth.md and RELEASE_NOTES.md for the Rust implementation.Pjotr Prins
* docs/gn-auth.md: Document the GN2 public>0 AND confidentiality<1 threshold (the test for whether data needs a gn-auth authorization call at all, versus just a valid token), the now-implemented src/auth.rs mechanism, and the four gn-auth bugs found and fixed while building it. * RELEASE_NOTES.md: Likewise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04doc: Add docs/gn-auth.md, how to call the gn-auth service.Pjotr Prins
* docs/gn-auth.md: New file. Explains how a client (genenetwork4) should authenticate callers against gn-auth: tokens are RS256 JWTs, verified locally via the JWKS at GET /auth/public-jwks (mirroring genenetwork3's gn3/oauth2/jwks.py, with two improvements -- cache and periodically refresh the key set instead of refetching per request, and check exp/nbf/iss, not just the signature); resource-scoped authorization checks (e.g. "can this token's user do X on InbredSet Y") via /auth/resource/populations/resource-id/... and /auth/resource/authorisation, mirroring genenetwork3's gn3/api/case_attributes.py:required_access. Documents why POST /auth/introspect should not be used yet (unconfigured client allow-list, no client ever created with the client_type it requires, and JWT tokens are never persisted for it to find). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04doc: Add ProbeSetXRef section to genenetwork_datatypes.md.Pjotr Prins
* docs/genenetwork_datatypes.md (ProbeSetXRef): New section, the expression (mRNA) counterpart to the PublishXRef section above: ProbeSetXRef's own fields, its relationships to ProbeSetFreeze/ ProbeFreeze/InbredSet/Tissue, ProbeSet, and ProbeSetData/ProbeSetSE/ Strain, and the lack of a declared uniqueness constraint on (ProbeSetFreezeId, ProbeSetId). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04doc: Add PublishXRef section; sharpen SKILL.md with new lessons.Pjotr Prins
* docs/genenetwork_datatypes.md (PublishXRef): New section, covering PublishXRef's own fields and its relationships to Phenotype, Publication, PublishData/PublishSE/NStrain and Strain, based on topics/systems/mariadb/precompute-publishdata.gmi. * docs/SKILL.md (mysql crate): Add notes on aliasing joined columns to match FromRow's UpperCamelCase conversion instead of reaching for #[mysql(rename = "...")], and on mysql::Pool::new's expensive default PoolConstraints (eagerly opens 10 connections) making it unsafe to call per request. (Testing): Add timing DB-backed test requests through a shared helper, since that's what actually caught the Pool::new issue above. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-03doc: Sharpen SKILL.md's rustc-mismatch section with what actually worked.Pjotr Prins
* docs/SKILL.md (Guix packaging): Replace the generic "pin crates down or bump rust" note with the concrete sequence used to fix this in genenetwork4: one-crate-at-a-time --precise pins (and why re-running generate-lockfile undoes them), the mysql_common let-chains/is_multiple_of case where no patch version within a minor line helps and the dependent itself needs downgrading, and querying the sparse crates.io index directly to compare MSRVs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-03doc: Add docs/SKILL.md, reusable Rust/Axum/MariaDB API practices.Pjotr Prins
* docs/SKILL.md: New file. Covers the mysql crate's feature/FromRow quirks, the database-name-vs-code-name convention, the ApiError/ request-id error shape, utoipa OpenAPI setup, the two-suite test pattern, Guix packaging traps, and commit style, as practiced building this repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-03doc: Add InbredSet section, pad tables for readability.Pjotr Prins
* docs/genenetwork_datatypes.md (Species)[Fields]: Pad table columns with spaces so pipe characters line up in a plain editor. (InbredSet): New section. Field semantics (sourced from the live schema and genenetwork2/gn2/base/data_set/datasetgroup.py's MappingMethodId mapping), relationships to PublishFreeze/GenoFreeze/ ProbeFreeze/StrainXRef, known quirks, and a survey of existing implementations across GeneNetwork2, GeneNetwork3, gn-guile and this repo. States "Group" as the recommended alias for InbredSet, backed by GeneNetwork2's /api/v_pre1/groups and /api/v_pre1/group/<name> routes, the DatasetGroup class docstring, the group_name variable convention used throughout GeneNetwork2 and GeneNetwork3, and the "Groups" heading in gn-hacking-documentation.gmi; notes "RISet" as a secondary, more genetics-specific alias that is a slight misnomer for InbredSet rows whose GeneticType is not an actual recombinant inbred set.
2026-10-03doc: Add docs/genenetwork_datatypes.md, Species section.Pjotr Prins
* docs/genenetwork_datatypes.md: New file. Species table field semantics (sourced from factory/docs/gn-gemtext-threads/topics/ gn-uploader/adding-species.gmi and this repo's own docs/mariadb_db-webqtl.md schema dump), its relationships to InbredSet, Strain, Chr_Length and GeneIDXRef, its role in GN's authorization and RDF classification scheme, known quirks (mouse-centric legacy genotype columns, undocumented FamilyOrderId/ OrderId, no UI for adding a species), and a survey of existing Species-related implementations across GeneNetwork2, GeneNetwork3, gn-guile and this repo.
2026-10-03doc: Add db_webqtl MariaDB schema dump.Pjotr Prins
* docs/mariadb_db-webqtl.md: New file. CREATE TABLE statement for each of the 112 tables in db_webqtl, dumped with mysqldump --no-data and grouped under a linked table of contents.