| Age | Commit message (Collapse) | Author |
|
A user reported that the token was fine and the service was up, but
"everything backed by the relational database is failing" -- while
`/search` (Xapian) and `/version` answered. By the time I looked, every
data endpoint answered 200 and `/version` said `database: reachable`, and
there was **no record anywhere** of what they had hit: the service logged
nothing at all, so a caller's report could not be matched to a cause, and
`/version`'s `database` is a *now* probe that says nothing about the
minute before.
Two things now make that diagnosable:
* every 5xx is written to stderr as one JSON line -- status, code,
operation_id, **request_id**, took_ms and a truncated detail -- so
"500 at 12:31, request id ..." can be looked up. 4xx stay unlogged: they
are the caller's business, and a 401/422 stream would drown this;
* the last database failure is remembered (`db::record_failure`, called
from `ApiError::from_db`, the one place every database error passes
through) and `/version` reports it in `notes` for ten minutes, with its
kind (`unavailable`, `connection_lost`, `query`) and the driver's
message. That is the gap that misled the user: a healthy probe beside
failing endpoints looks like an auth problem from outside.
`is_connection_loss` gained a `_detail` form so the caller that already
has the rendered message does not render it twice.
* src/error.rs: the log line, and recording failures by kind;
* src/db.rs: `FailureKind`, `Failure`, `record_failure`, `recent_failure`,
and two unit tests (a failure is remembered and ages out; a driver
message is classified correctly);
* src/lib.rs: the `/version` note, absent unless something failed recently.
Checked against a deliberately broken `SQL_URI` on a scratch instance: the
data endpoint answers `503 db_unavailable`, the log carries the
request id, and `/version` shows the note while still answering 200.
|
|
Every database-backed endpoint 500'd with `CodecError { IO error: Broken
pipe }` while the process answered /version quite happily. The cause was
ours: `pool()` turns off the driver's `check_health` (a ping at checkout,
dead connections replaced) to save a round trip -- sound against a
database 130 ms away, worth nothing against loopback, and it costs the
pool its self-healing. One server-side close of an idle session then
becomes one failed request per dead socket: ten in a row, until a
restart.
* src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping;
`is_connection_loss` recognises the errors that mean the peer went away
(broken pipe, reset, went away, lost connection, closed);
`forget_pool` drops the cached pool so the next call rebuilds it;
`health` probes reachability, cached ten seconds. One pool static shared
by pool()/forget_pool (they had drifted into two).
* src/error.rs (ApiError::from_db): on a connection loss, forget the pool
-- the one place every database error passes through, so recovery does
not depend on each caller.
* src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a
service that answers while nothing behind it works is now visible.
* src/config.rs: the switch, with the trade written down.
* tests/db_recovery.rs: kills every pooled session from a second
connection, then asserts the next twelve requests all succeed -- the
scenario that produced the outage, now a regression test.
* tests/...: unit test for the classifier (its own IO error text and the
"this is a bad query, do not retry" cases).
* README.md ("Database connections"), docs/www/operator-checklist.md:
the knob, why it defaults on, and what /version reports.
|
|
* src/config.rs (DbMode, Config.db_mode, parse_db_mode): add
GN4_DB_MODE=dev|production in the layered config; dev runs with no
configured DB; fail-closed parse (a typo refuses).
* src/db.rs (DbError, pool, probe), src/error.rs (ApiError::from_db):
mint a typed 503 db_unavailable at the db::pool() choke point,
distinguishing Unavailable from Query.
* src/main.rs: loud no-DB startup line; production refuses to start
(exit non-zero) before axum::serve.
* src/routes/{accessions,datasets,groups,mapping_methods,publications,species,traits}.rs:
route every DB error through ApiError::from_db.
* src/bin/gn-indexer.rs: refuse to build an index when the DB is unreachable.
* tests/no_db.rs: pin no-DB degradation (search 200 partial, DB routes 503).
His ask (DM 2026-10-12): "Make it an option to switch it off for
development machines? I mean mariadb." The no-DB hardening is ours
(review-driven). Default GN4_DB_MODE left unflipped (=dev); the flip needs
owner sign-off.
|
|
Breaking: every success body is now {"data", "meta": {"state",
"took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only
`/` (plain text) and `/openapi.json` are unwrapped.
* src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers
take Ctx first and build the response with ctx.envelope(data), so
timing and the request id are never filled in by hand. `state` is
`complete` or `partial`; notes are server-authored only; `next` are
concrete relative hrefs (path_segment percent-encodes names).
* src/routes/*.rs, src/lib.rs (version): every handler returns
Envelope<..> with next links; /search is `partial` when the locus
lookup fails or a sorted 1000-hit window may be truncated, and
/publications/{pubmed_id}/traits when datasets were left out for
authorization.
* src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member.
* src/auth.rs (AuthToken): the 401 now carries request_id and took_ms
(it never had a request_id); Ctx stores RequestInfo in the request
extensions for it.
* tests: common::data / envelope_data parse every success body through
the envelope; tests/envelope.rs and test/python/test_envelope.py check
the shape, errors, and that every `next` link resolves;
tests/envelope_partial.rs checks the `partial` search without a
database; tests/openapi.rs fails if a JSON success response is not an
Envelope_* schema.
* README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document
the envelope and took_ms on errors.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
One commit, both halves of AMENDMENT E / the 2026-10-06 contract:
(i) The 422 invalid_query body gains the blessed typed shape. The
SearchError::InvalidQuery payload becomes {message, xapian_error_type},
threaded from every producer:
- native parser rejects (parse_query_checked): the ParseError's own
class, e.g. QueryParserError for Q51/Q52;
- the C3.4 numeric-range guard: QueryParserError ("Unknown range
operation", Q52's recorded production diagnostic);
- the C3.6 invalid-location guard: InvalidOperationError — the class
production actually threw for Q42;
- the gn3 forwarder: gn3's error_type field, with the bytes-repr leak
normalized (b'QueryParserError' -> QueryParserError — the binding-
report R1 / OPEN-16 wrapper-layer normalization), or the exception
class name itself for gn3's escaping-exception 500 bodies.
The route's engine_error builds the typed body inside the existing
ApiError envelope, under "error": {type: "invalid_query", message,
param: "query", detail: {xapian_error_type}} — the envelope's existing
fields (code, detail, ...) keep their meaning, so every current 422
(Q51/Q52 parser rejects, Q42's invalid-location class, mean:abc..def)
emits it and the harness's body["code"] grading is untouched.
The forwarder additionally maps gn3's malformed-processor-value 500s
(the escaping xapian exceptions — Q42's recorded
{"error": "InvalidOperationError", ...} body, the parity-metric
amendment's curl-verified ValueError) onto InvalidQuery, scoped to the
exception-class shape (a bare CamelCase ...Error name, or "Class:
message"): the blessed contract folds all malformed input into 422,
while prose 500 diagnostics ("gn3 exploded") keep the 502
backend_error class — pinned by both the rust and python route tests.
(ii) The harness comparator's Q42 mapping rule changes: recorded
production 500 -> expected 422 invalid_query. The recorded fixture
itself stays untouched as production truth (recorded_status 500,
served verbatim by the stub); only grade_error_class's Server500 arm
re-baselines, with its doc comments and the comparator unit tests.
Gate evidence (~/spikes/port-execution-r2/logs/c37-*): production
floor 54/54 PASS with the NEW rule — Q42 rec=500 gn4=422
code=invalid_query through the real route + forwarding engine + stub
(match200 49/49, clause-b 36/36, min J 1.000, parity-report.json
floor PASS); FULL pinned cargo test --release EXIT=0 (30 suites,
incl. the re-based comparator unit tests and the route unit tests
pinning the typed body's presence for invalid_query and absence for
backend errors); shape-class gate 52/52 unchanged; hermetic lib 23
green incl. the typed-payload assertions (Q51 message byte-identical,
QueryParserError classification). rustfmt --edition 2024 clean on all
nine touched files.
|
|
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs,
TestToken replacing DevToken), matching tests/get_test_token.rs and
test/python/test_get_test_token.py (renamed from
tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and
the GN_AUTH_DEV_* config keys keep their names -- they're about the
dev-token feature generically, not tied to this one route's path.
Every endpoint now requires a valid token, not just the
dataset_probeset*/traits* family: species(_by_name/_datasets),
groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all
gain an AuthToken parameter. /, /version, /openapi.json and
/get-test-token itself are the only exceptions -- a health check, a
version check, the spec describing all this, and the endpoint that
hands out a token in the first place (which would be circular if it
needed one).
Rather than repeating security(("bearer_token" = [])) on every
#[utoipa::path], ApiDoc now declares it once as the document's default
security(...), and the three exceptions besides /get-test-token (/,
/version) override it with their own empty security() -- standard
OpenAPI semantics (an operation's own security, even empty, overrides
the document default; an absent one inherits it). error::Unauthorized,
a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the
shared 401 every gated handler now references bare in its own
responses(...) list instead of writing out the same
(status = 401, description = "...", body = ApiError) tuple by hand.
Verified the resolved security per operation in a live /openapi.json.
tests/common::auth_token() (Rust) and the new
test/python/auth_helpers.auth_token() (Python, deduplicating what four
files previously each defined locally) now try this server's own
POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN --
every previously-ungated test file gains a token check and switches to
get_with_token; the whole suite, both languages, now runs with zero
manually-provisioned tokens given GENENETWORK4_DEV_MODE=true
GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains
EXTRA_ENV so each Python test class can start its own server with dev
mode on. Verified both configured (auto-fetch) and unconfigured
(GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g.
the Guix build sandbox) paths.
README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for
the rename and the new blanket token requirement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* Cargo.toml (jsonwebtoken, ureq): New dependencies, RS256 JWT
verification and gn-auth's HTTP API respectively. ureq pulls in ring
for RSA verification, needing a C compiler at build time -- already
provided by cargo-build-system's standard inputs, nothing added to
guix.scm for it. Promote serde_json from dev-dependencies to
dependencies; src/auth.rs needs it outside tests now.
(mysql-common-derive, simple_asn1, time): Pin to the newest versions
still compatible with the Guix-packaged rustc 1.85.1 (same trap as
before: ureq's dependency chain pulls darling 0.23/time 0.3.46+, both
needing rustc 1.88).
* rust-package.scm: Regenerate with `guix import crate -f Cargo.lock
genenetwork4`.
* guix.scm (genenetwork4-git)[native-inputs]: Update crate list to
match.
* src/auth.rs (AuthToken): New extractor. Verifies a bearer token's
RS256 signature and standard time claims (exp/nbf, checked only when
present -- gn-auth's own validator treats them as optional too)
against gn-auth's JWKS, cached and refreshed periodically
(OnceLock<Mutex<HashMap<...>>>, same shape as db::pool()) rather than
fetched per request.
(is_publicly_readable): New function. Matches GeneNetwork2's own
threshold (public > 0 AND confidentiality < 1, used throughout
wqflask/api/router.py) for whether a dataset needs an authorization
check at all; InbredSet/group rows carry a public column too, but GN2
never filters on it, so groups/species endpoints need no gating.
(authorise): New function. Wraps gn-auth's POST
/auth/resource/authorisation, caching the per-(user, resource) answer
briefly (AUTHORISATION_TTL) so repeated requests for the same resource
by the same user don't each cost a round trip. Not yet called from any
route -- see RELEASE_NOTES.md.
* src/config.rs (Config)[auth_server_url]: New field, AUTH_SERVER_URL
config key, matching GeneNetwork3's own key of the same name.
* src/error.rs (ApiError): Derive Debug; needed by tests/auth.rs's
.unwrap() on a Result<_, ApiError>, and every error type should
implement it regardless.
* src/lib.rs: Register the new auth module.
* tests/auth.rs: New file. Runs AuthToken verification and authorise()
(including its cache) against a real gn-auth instance; set
GN4_TEST_AUTH_TOKEN/GN4_TEST_RESOURCE_ID (see docs/gn-auth.md), skips
otherwise.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Line-wrapping and canonical import ordering only (verified with
git diff -w). Establishes rustfmt's output as the canonical style
before the search/correlation work lands.
|
|
* Cargo.toml (dependencies): Add tower-http, features request-id and
util.
* src/error.rs: New file. ApiError, modeled on RFC 7807 (Problem
Details for HTTP APIs) plus code, operation_id, spec_url, request_id
and next. type and spec_url are relative paths since this service has
no fixed public domain. Implements axum::response::IntoResponse.
* src/lib.rs (app): Add SetRequestIdLayer and PropagateRequestIdLayer,
assigning each request an x-request-id and copying it onto the
response.
(request_id_string): New function.
(species, species_by_name): Take an Extension<RequestId> parameter.
Return Result<_, ApiError> instead of Result<_, (StatusCode, String)>.
(species_by_name): On no match, set ApiError's next to /species, so a
caller with a wrong species name can discover the valid ones.
* tests/species_by_name.rs (species_by_name_unknown_species_returns_404):
Check the full ApiError body (status, code, operation_id, spec_url,
next) and that request_id matches the x-request-id response header,
instead of only the status code.
* test/python/test_species_by_name.py
(test_species_by_name_unknown_species_returns_404): Likewise.
* README.md: Add an Error handling section documenting the ApiError
shape and each field.
* rust-package.scm: Regenerate with guix import crate -f Cargo.lock
genenetwork4. Adds crate-source definitions for tower-http and its
new transitive dependencies.
* guix.scm (genenetwork4-git)[native-inputs]: Add rust-tower-http-0.7.1,
rust-getrandom-0.4.3 and rust-r-efi-6.0.0.
|