about summary refs log tree commit diff
path: root/tests/common
AgeCommit message (Collapse)Author
8 daysbaskets: a user's own column of values, in their own SQLite filePjotr
The next phase is changing data, and the constraint is that MariaDB is never written to. A basket is the storage half of that: take a trait's values, edit some of them (or build a covariate column that does not exist), keep the column under a name, and map that instead. * POST /baskets stores a complete column; GET /baskets lists yours; GET /baskets/{name} its metadata; GET /baskets/{name}/values the cells; PATCH /baskets/{name} replaces the column with a new revision; DELETE removes it. * POST /compute/qtl2 takes "basket": "<name>" in place of `samples`, and both at once is a 422 rather than a silent choice. The mapping code below it is untouched -- a basket resolves to the same (name, value) pairs a client would have posted. * One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600, WAL, busy_timeout. `GET /baskets` only ever lists the caller's own; the file name is a hash, so a directory listing does not name users. * The full column is stored, not a diff -- the caller has the original beside it, and a mapping needs one complete column. * Text with a parsed copy beside it: raw (what was typed), value (the number or NULL), status (ok/missing/bad). "NA" and 0 stay different claims, and a column with something that is neither a number nor a missing marker (NA, x, -, ., ?) is refused with the samples named -- silently dropping it would change a trait without saying so. * A revision and a digest: an edit bumps the revision and changes the digest, and a mapping reply records both ("Basket 'wbc-edit' revision 2 (digest 9559a70e...)"), so two runs on one name cannot look alike. A result cache must key on the digest, not the name. The loop, end to end on BXD trait 10001: 34 values posted as a basket, one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2 -- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the revision and digest. tests/baskets.rs walks that loop (create, list, values, patch, map by name, delete) and the refusals: a non-number named in the error, a name that cannot become a path, an unknown basket, and basket-plus-samples. Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest changing on edit, per-user isolation, and name validation. rqtl/gemma take the same 10-line change next -- their `samples` shapes differ, so they are a separate commit rather than a rushed one.
2026-10-14rustfmt: apply rustfmt across the treeAlexander_Kabui
Pure formatting sweep (import ordering, line wrapping); the tree had accumulated rustfmt drift. Separated from any semantic change so it stays reviewable and revertable on its own.
2026-10-11gemma: add POST /compute/gemma, backed by the real gemma-wrapper/GEMMAPjotr
Maps a QTL from a genotype file, a SNP annotation file and trait values, by running gemma-wrapper's K-matrix step then its LMM association step, both synchronously (axum/tokio awaits the subprocess directly, no job queue needed) and returning the parsed .assoc.txt rows in the standard envelope. Both steps use a bare --loco (gemma-wrapper derives the chromosome list from the SNP annotation file). This was arrived at empirically, not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real against the BXD fixtures, - GeneNetwork3's gn3/computations/gemma.py composes a GWA command that always keeps "-gk" alongside "-lmm"; real GEMMA rejects that combination outright ("only one of -gk ... -lmm ... options is allowed"), so none of its gwa-compute/k-gwa-compute endpoints can produce output. Unsurprising: tests/integration/test_gemma.py is entirely @unittest.skip, so this was never exercised. - GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly omits "-gk" from the GWA step, but its non-LOCO branch fails too, the same way my own first (non-LOCO) attempt did: gemma-wrapper only ever archives a non-LOCO kinship matrix into a .tar.xz, with no loose file at the path its own non-LOCO GWA code assumes ("Enforce failed for file_kin ... open file"). Only --loco, which gemma-wrapper's own USAGE banner documents and which does unpack that archive, actually works. A --loco run's .assoc.txt files (one per chromosome) are themselves only ever inside gemma-wrapper's own output archive, never loose, so gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated dependencies) to extract them before concatenating all chromosomes' rows into the response. guix.scm now depends on (gn packages gemma) from guix-bioinformatics for gemma-wrapper (propagated-input: it's invoked as a subprocess, not linked, and this makes it land in a consuming profile too, same as GeneNetwork3's and GeneNetwork2's own packages do it). That dependency needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not build at all before that fix. Verified end to end: `cargo test` passes in full, including a real (non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD fixtures copied from genenetwork3/tests/test_data, producing exactly 7320 association rows (one per marker in BXD_snps.txt) in ~19s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-08* src/routes/dev.rs: POST /get-test-token answers local callers only, not ↵Pjotr
tied to dev mode. The endpoint no longer depends on GENENETWORK4_DEV_MODE or on the bind address. It decides per request whether to answer. * src/routes/dev.rs (get_test_token): answers a caller only when the TCP peer (axum ConnectInfo) is in 127.0.x.x (or ::1, or an IPv4-mapped 127.x) and the request carries no X-Forwarded-For, Forwarded or X-Real-IP header (a reverse proxy on the same host would otherwise make every client look local). Anything else, including an unknown peer, is a 403. Local is never established from a header. * src/main.rs: serve with into_make_service_with_connect_info, which provides the peer address. * src/config.rs: remove dev_mode, GENENETWORK4_DEV_MODE and dev_token_enabled(). Add GENENETWORK4_TEST_TOKEN (on by default); when it is not a clearly affirmative value (true/1/yes/on) the route is not registered: a 404, absent from /openapi.json. It fails closed, so a typo ("flase") turns the endpoint off. * src/lib.rs: register the route unless that switch is off. * tests: tests/get_test_token.rs rewritten (remote, unknown and proxied peers refused; 127.x and ::1 answered; always documented by default); tests/get_test_token_off.rs for the switch (own process: it sets an environment variable); common::post_from supplies a peer address to the in-process requests. Python: test_get_test_token.py rewritten, TOKEN_ENV is now empty (no setting needed). * README.md, docs/gn-auth.md, docs/SKILL.md, RELEASE_NOTES.md: document the check, the switch, and "Behind nginx or in a container" (a bridge container sees the gateway as the peer; a host-network container behind nginx sees 127.0.0.1, so block the path in nginx, set X-Forwarded-For, or switch the endpoint off). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/response.rs: wrap every success response in one envelope.Pjotr
Breaking: every success body is now {"data", "meta": {"state", "took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only `/` (plain text) and `/openapi.json` are unwrapped. * src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers take Ctx first and build the response with ctx.envelope(data), so timing and the request id are never filled in by hand. `state` is `complete` or `partial`; notes are server-authored only; `next` are concrete relative hrefs (path_segment percent-encodes names). * src/routes/*.rs, src/lib.rs (version): every handler returns Envelope<..> with next links; /search is `partial` when the locus lookup fails or a sorted 1000-hit window may be truncated, and /publications/{pubmed_id}/traits when datasets were left out for authorization. * src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member. * src/auth.rs (AuthToken): the 401 now carries request_id and took_ms (it never had a request_id); Ctx stores RequestInfo in the request extensions for it. * tests: common::data / envelope_data parse every success body through the envelope; tests/envelope.rs and test/python/test_envelope.py check the shape, errors, and that every `next` link resolves; tests/envelope_partial.rs checks the `partial` search without a database; tests/openapi.rs fails if a JSON success response is not an Envelope_* schema. * README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document the envelope and took_ms on errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-05Rename /dev/token to /get-test-token; require a token everywherePjotr Prins
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs, TestToken replacing DevToken), matching tests/get_test_token.rs and test/python/test_get_test_token.py (renamed from tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and the GN_AUTH_DEV_* config keys keep their names -- they're about the dev-token feature generically, not tied to this one route's path. Every endpoint now requires a valid token, not just the dataset_probeset*/traits* family: species(_by_name/_datasets), groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all gain an AuthToken parameter. /, /version, /openapi.json and /get-test-token itself are the only exceptions -- a health check, a version check, the spec describing all this, and the endpoint that hands out a token in the first place (which would be circular if it needed one). Rather than repeating security(("bearer_token" = [])) on every #[utoipa::path], ApiDoc now declares it once as the document's default security(...), and the three exceptions besides /get-test-token (/, /version) override it with their own empty security() -- standard OpenAPI semantics (an operation's own security, even empty, overrides the document default; an absent one inherits it). error::Unauthorized, a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the shared 401 every gated handler now references bare in its own responses(...) list instead of writing out the same (status = 401, description = "...", body = ApiError) tuple by hand. Verified the resolved security per operation in a live /openapi.json. tests/common::auth_token() (Rust) and the new test/python/auth_helpers.auth_token() (Python, deduplicating what four files previously each defined locally) now try this server's own POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN -- every previously-ungated test file gains a token check and switches to get_with_token; the whole suite, both languages, now runs with zero manually-provisioned tokens given GENENETWORK4_DEV_MODE=true GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains EXTRA_ENV so each Python test class can start its own server with dev mode on. Verified both configured (auto-fetch) and unconfigured (GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g. the Guix build sandbox) paths. README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for the rename and the new blanket token requirement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-05Add POST /dev/token, a dev-only gn-auth token issuerPjotr Prins
Calls gn-auth's own password grant (POST /auth/token) for gn-auth's well-known development user and returns the result, so local testing no longer needs a manual curl/gn-auth round trip of its own. Only ever registered -- not just rejected at request time, genuinely absent from the router and from /openapi.json -- when two new Config fields are both set: dev_mode (GENENETWORK4_DEV_MODE) and a loopback bind (GENENETWORK4_BIND, also new). Both are config, not something genenetwork4 infers from its actual network exposure, by design: dev mode alone, or a loopback bind alone, each leave it disabled, and the check (Config::dev_token_enabled) is a plain function so the condition itself has direct unit test coverage rather than only being exercised indirectly through the route. src/main.rs now binds to config.bind instead of a hardcoded "0.0.0.0", via a proper SocketAddr rather than string formatting -- the latter would mishandle an IPv6 bind address like "::1" (needs brackets). GN_AUTH_DEV_CLIENT_ID/_CLIENT_SECRET/_USERNAME/_PASSWORD default to gn-auth's own published dev fixtures (flask init-dev-clients, gn_auth/wsgi.py), so this works unconfigured against an unmodified dev gn-auth instance -- confirmed live, including using the returned token against a real gated endpoint (/traits/BXD/10001), not just checking the token response's shape. tests/dev_token.rs covers all four combinations of the two settings in one sequential test function (mutating the same two env vars from more than one #[tokio::test] in a file would race, since they run concurrently by default) plus the /openapi.json visibility check. test/python/test_dev_token.py mirrors this using separate test classes instead, which is safe here since each spawns its own server subprocess with its own environment (ServerTestCase gains EXTRA_ENV for this). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Gate GET /datasets/{dataset}/probesets/{probeset} on gn-authPjotr Prins
Require a valid gn-auth bearer token on this endpoint unconditionally. For a dataset that isn't publicly readable, check the group:resource:view-resource privilege via a new auth::authorise_trait, the "Dataset::Trait"-keyed equivalent of authorise (POST /auth/data/authorisation instead of /auth/resource/authorisation) -- gn-auth resolves the resource itself from the dataset name, so no separate resource-id lookup is needed here. A trait never linked to a gn-auth resource (gn-auth answers 404) is treated as "no privileges", not a request failure. Denied requests get a 403; a failure to reach/parse gn-auth's answer gets a 502, distinct from both that and this service's own 500s. is_publicly_readable now also takes the dataset's CreateTime and treats anything older than 5 years as public regardless of its public/confidentiality flags -- a confidentiality embargo isn't meant to last forever. No admin/sysadmin bypass is implemented. Register a "bearer_token" OpenAPI security scheme (src/lib.rs) so the new security(...) annotation on this endpoint's #[utoipa::path] has something to point at. tests/dataset_probeset_info.rs and its Python mirror gain cases for the missing-token (401) and denied-but-existing (403) outcomes, run against a real gn-auth instance set up as described in docs/gn-auth.md; the 403 case uses UTHSC_BXD_Cochlea_0225, a real non-public, non-embargoed dataset never linked into gn-auth. tests/common gains get_with_token and auth_server_available/auth_token helpers for this. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /groups/{group}/traits/{trait_id}; time DB tests; share pool.Pjotr Prins
* src/db.rs (TraitInfo): New struct. Joins PublishXRef, Phenotype and Publication; LEFT JOIN on the latter two since a handful of PublishXRef rows have no matching row. SQL column aliases are written to match mysql_common's UpperCamelCase conversion exactly, so no field needs a #[mysql(rename = "...")] override. (get_group_trait_info): New function. (pool): New function. mysql::Pool::new eagerly opens its default 10 connections before returning (about 7s against this database); every query function here used to call it per request, paying that cost on every single API call. Builds the Pool once behind a OnceLock<Mutex<Option<Pool>>> and shares the cheap, Arc-backed handle from then on (OnceLock::get_or_init can't be used alone since Pool::new is fallible and get_or_try_init is still unstable). (get_all_species, get_species_by_name, get_all_groups, get_group_by_name, get_group_strains, get_group_trait_values, get_group_trait_info): Use it instead of building their own Pool. * src/lib.rs (GroupTraitLookup): Generalize over the found value (was Vec<db::TraitValue>-specific), so group_trait_info can share it with group_trait_values. (get_group_trait_info_result): New function. (group_trait_info): New handler for GET /groups/{group}/traits/{trait_id}. (app): Register it. * tests/common/mod.rs: New file. get() issues a GET against genenetwork4::app() and warns on stderr if it takes over a second -- this is what surfaced the Pool::new cost above. * tests/species.rs, tests/species_by_name.rs, tests/groups.rs, tests/group_by_name.rs, tests/group_strains.rs, tests/group_trait_values.rs: Use tests/common::get instead of building each request by hand. * tests/group_trait_info.rs, test/python/test_group_trait_info.py: New files. * tests/openapi.rs, test/python/test_openapi.py: Check the new path is in the generated spec. * README.md (GET /groups/{group}/traits/{trait_id}): Document it. (Tests): List the new test files; document tests/common::get's timing warnings and how to see them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Fix DB tests: probe SQL_URI credentials, not just the portAlexander_Kabui
The old skip check connected to 127.0.0.1:3306, so tests ran (and failed with 500) whenever a server was listening but the configured credentials were rejected. tests/common/mod.rs now attempts a real connection with the configured SQL_URI; tests skip only when the database is genuinely unusable, and run for real otherwise.