| Age | Commit message (Collapse) | Author |
|
The next phase is changing data, and the constraint is that MariaDB is
never written to. A basket is the storage half of that: take a trait's
values, edit some of them (or build a covariate column that does not
exist), keep the column under a name, and map that instead.
* POST /baskets stores a complete column; GET /baskets lists yours;
GET /baskets/{name} its metadata; GET /baskets/{name}/values the
cells; PATCH /baskets/{name} replaces the column with a new revision;
DELETE removes it.
* POST /compute/qtl2 takes "basket": "<name>" in place of `samples`,
and both at once is a 422 rather than a silent choice. The mapping
code below it is untouched -- a basket resolves to the same
(name, value) pairs a client would have posted.
* One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600,
WAL, busy_timeout. `GET /baskets` only ever lists the caller's own;
the file name is a hash, so a directory listing does not name users.
* The full column is stored, not a diff -- the caller has the original
beside it, and a mapping needs one complete column.
* Text with a parsed copy beside it: raw (what was typed), value (the
number or NULL), status (ok/missing/bad). "NA" and 0 stay different
claims, and a column with something that is neither a number nor a
missing marker (NA, x, -, ., ?) is refused with the samples named --
silently dropping it would change a trait without saying so.
* A revision and a digest: an edit bumps the revision and changes the
digest, and a mapping reply records both ("Basket 'wbc-edit'
revision 2 (digest 9559a70e...)"), so two runs on one name cannot
look alike. A result cache must key on the digest, not the name.
The loop, end to end on BXD trait 10001: 34 values posted as a basket,
one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2
-- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the
revision and digest.
tests/baskets.rs walks that loop (create, list, values, patch, map by
name, delete) and the refusals: a non-number named in the error, a name
that cannot become a path, an unknown basket, and basket-plus-samples.
Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest
changing on edit, per-user isolation, and name validation.
rqtl/gemma take the same 10-line change next -- their `samples` shapes
differ, so they are a separate commit rather than a rushed one.
|
|
Pure formatting sweep (import ordering, line wrapping); the tree
had accumulated rustfmt drift. Separated from any semantic change
so it stays reviewable and revertable on its own.
|
|
Maps a QTL from a genotype file, a SNP annotation file and trait
values, by running gemma-wrapper's K-matrix step then its LMM
association step, both synchronously (axum/tokio awaits the subprocess
directly, no job queue needed) and returning the parsed .assoc.txt
rows in the standard envelope.
Both steps use a bare --loco (gemma-wrapper derives the chromosome
list from the SNP annotation file). This was arrived at empirically,
not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real
against the BXD fixtures,
- GeneNetwork3's gn3/computations/gemma.py composes a GWA command that
always keeps "-gk" alongside "-lmm"; real GEMMA rejects that
combination outright ("only one of -gk ... -lmm ... options is
allowed"), so none of its gwa-compute/k-gwa-compute endpoints can
produce output. Unsurprising: tests/integration/test_gemma.py is
entirely @unittest.skip, so this was never exercised.
- GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly
omits "-gk" from the GWA step, but its non-LOCO branch fails too,
the same way my own first (non-LOCO) attempt did: gemma-wrapper only
ever archives a non-LOCO kinship matrix into a .tar.xz, with no
loose file at the path its own non-LOCO GWA code assumes
("Enforce failed for file_kin ... open file"). Only --loco, which
gemma-wrapper's own USAGE banner documents and which does unpack
that archive, actually works.
A --loco run's .assoc.txt files (one per chromosome) are themselves
only ever inside gemma-wrapper's own output archive, never loose, so
gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated
dependencies) to extract them before concatenating all chromosomes'
rows into the response.
guix.scm now depends on (gn packages gemma) from guix-bioinformatics
for gemma-wrapper (propagated-input: it's invoked as a subprocess, not
linked, and this makes it land in a consuming profile too, same as
GeneNetwork3's and GeneNetwork2's own packages do it). That dependency
needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see
that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not
build at all before that fix.
Verified end to end: `cargo test` passes in full, including a real
(non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD
fixtures copied from genenetwork3/tests/test_data, producing exactly
7320 association rows (one per marker in BXD_snps.txt) in ~19s.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
tied to dev mode.
The endpoint no longer depends on GENENETWORK4_DEV_MODE or on the bind
address. It decides per request whether to answer.
* src/routes/dev.rs (get_test_token): answers a caller only when the TCP
peer (axum ConnectInfo) is in 127.0.x.x (or ::1, or an IPv4-mapped
127.x) and the request carries no X-Forwarded-For, Forwarded or
X-Real-IP header (a reverse proxy on the same host would otherwise make
every client look local). Anything else, including an unknown peer, is a
403. Local is never established from a header.
* src/main.rs: serve with into_make_service_with_connect_info, which
provides the peer address.
* src/config.rs: remove dev_mode, GENENETWORK4_DEV_MODE and
dev_token_enabled(). Add GENENETWORK4_TEST_TOKEN (on by default); when it
is not a clearly affirmative value (true/1/yes/on) the route is not
registered: a 404, absent from /openapi.json. It fails closed, so a
typo ("flase") turns the endpoint off.
* src/lib.rs: register the route unless that switch is off.
* tests: tests/get_test_token.rs rewritten (remote, unknown and proxied
peers refused; 127.x and ::1 answered; always documented by default);
tests/get_test_token_off.rs for the switch (own process: it sets an
environment variable); common::post_from supplies a peer address to the
in-process requests. Python: test_get_test_token.py rewritten, TOKEN_ENV
is now empty (no setting needed).
* README.md, docs/gn-auth.md, docs/SKILL.md, RELEASE_NOTES.md: document
the check, the switch, and "Behind nginx or in a container" (a bridge
container sees the gateway as the peer; a host-network container behind
nginx sees 127.0.0.1, so block the path in nginx, set X-Forwarded-For,
or switch the endpoint off).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Breaking: every success body is now {"data", "meta": {"state",
"took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only
`/` (plain text) and `/openapi.json` are unwrapped.
* src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers
take Ctx first and build the response with ctx.envelope(data), so
timing and the request id are never filled in by hand. `state` is
`complete` or `partial`; notes are server-authored only; `next` are
concrete relative hrefs (path_segment percent-encodes names).
* src/routes/*.rs, src/lib.rs (version): every handler returns
Envelope<..> with next links; /search is `partial` when the locus
lookup fails or a sorted 1000-hit window may be truncated, and
/publications/{pubmed_id}/traits when datasets were left out for
authorization.
* src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member.
* src/auth.rs (AuthToken): the 401 now carries request_id and took_ms
(it never had a request_id); Ctx stores RequestInfo in the request
extensions for it.
* tests: common::data / envelope_data parse every success body through
the envelope; tests/envelope.rs and test/python/test_envelope.py check
the shape, errors, and that every `next` link resolves;
tests/envelope_partial.rs checks the `partial` search without a
database; tests/openapi.rs fails if a JSON success response is not an
Envelope_* schema.
* README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document
the envelope and took_ms on errors.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs,
TestToken replacing DevToken), matching tests/get_test_token.rs and
test/python/test_get_test_token.py (renamed from
tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and
the GN_AUTH_DEV_* config keys keep their names -- they're about the
dev-token feature generically, not tied to this one route's path.
Every endpoint now requires a valid token, not just the
dataset_probeset*/traits* family: species(_by_name/_datasets),
groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all
gain an AuthToken parameter. /, /version, /openapi.json and
/get-test-token itself are the only exceptions -- a health check, a
version check, the spec describing all this, and the endpoint that
hands out a token in the first place (which would be circular if it
needed one).
Rather than repeating security(("bearer_token" = [])) on every
#[utoipa::path], ApiDoc now declares it once as the document's default
security(...), and the three exceptions besides /get-test-token (/,
/version) override it with their own empty security() -- standard
OpenAPI semantics (an operation's own security, even empty, overrides
the document default; an absent one inherits it). error::Unauthorized,
a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the
shared 401 every gated handler now references bare in its own
responses(...) list instead of writing out the same
(status = 401, description = "...", body = ApiError) tuple by hand.
Verified the resolved security per operation in a live /openapi.json.
tests/common::auth_token() (Rust) and the new
test/python/auth_helpers.auth_token() (Python, deduplicating what four
files previously each defined locally) now try this server's own
POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN --
every previously-ungated test file gains a token check and switches to
get_with_token; the whole suite, both languages, now runs with zero
manually-provisioned tokens given GENENETWORK4_DEV_MODE=true
GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains
EXTRA_ENV so each Python test class can start its own server with dev
mode on. Verified both configured (auto-fetch) and unconfigured
(GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g.
the Guix build sandbox) paths.
README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for
the rename and the new blanket token requirement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Calls gn-auth's own password grant (POST /auth/token) for gn-auth's
well-known development user and returns the result, so local testing
no longer needs a manual curl/gn-auth round trip of its own.
Only ever registered -- not just rejected at request time, genuinely
absent from the router and from /openapi.json -- when two new Config
fields are both set: dev_mode (GENENETWORK4_DEV_MODE) and a loopback
bind (GENENETWORK4_BIND, also new). Both are config, not something
genenetwork4 infers from its actual network exposure, by design: dev
mode alone, or a loopback bind alone, each leave it disabled, and the
check (Config::dev_token_enabled) is a plain function so the condition
itself has direct unit test coverage rather than only being exercised
indirectly through the route.
src/main.rs now binds to config.bind instead of a hardcoded "0.0.0.0",
via a proper SocketAddr rather than string formatting -- the latter
would mishandle an IPv6 bind address like "::1" (needs brackets).
GN_AUTH_DEV_CLIENT_ID/_CLIENT_SECRET/_USERNAME/_PASSWORD default to
gn-auth's own published dev fixtures (flask init-dev-clients,
gn_auth/wsgi.py), so this works unconfigured against an unmodified dev
gn-auth instance -- confirmed live, including using the returned token
against a real gated endpoint (/traits/BXD/10001), not just checking
the token response's shape.
tests/dev_token.rs covers all four combinations of the two settings in
one sequential test function (mutating the same two env vars from more
than one #[tokio::test] in a file would race, since they run
concurrently by default) plus the /openapi.json visibility check.
test/python/test_dev_token.py mirrors this using separate test classes
instead, which is safe here since each spawns its own server
subprocess with its own environment (ServerTestCase gains EXTRA_ENV for
this).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Require a valid gn-auth bearer token on this endpoint unconditionally.
For a dataset that isn't publicly readable, check the
group:resource:view-resource privilege via a new auth::authorise_trait,
the "Dataset::Trait"-keyed equivalent of authorise (POST
/auth/data/authorisation instead of /auth/resource/authorisation) --
gn-auth resolves the resource itself from the dataset name, so no
separate resource-id lookup is needed here. A trait never linked to a
gn-auth resource (gn-auth answers 404) is treated as "no privileges",
not a request failure. Denied requests get a 403; a failure to
reach/parse gn-auth's answer gets a 502, distinct from both that and
this service's own 500s.
is_publicly_readable now also takes the dataset's CreateTime and treats
anything older than 5 years as public regardless of its
public/confidentiality flags -- a confidentiality embargo isn't meant
to last forever. No admin/sysadmin bypass is implemented.
Register a "bearer_token" OpenAPI security scheme (src/lib.rs) so the
new security(...) annotation on this endpoint's #[utoipa::path] has
something to point at.
tests/dataset_probeset_info.rs and its Python mirror gain cases for the
missing-token (401) and denied-but-existing (403) outcomes, run against
a real gn-auth instance set up as described in docs/gn-auth.md; the
403 case uses UTHSC_BXD_Cochlea_0225, a real non-public, non-embargoed
dataset never linked into gn-auth. tests/common gains get_with_token
and auth_server_available/auth_token helpers for this.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* src/db.rs (TraitInfo): New struct. Joins PublishXRef, Phenotype and
Publication; LEFT JOIN on the latter two since a handful of
PublishXRef rows have no matching row. SQL column aliases are written
to match mysql_common's UpperCamelCase conversion exactly, so no field
needs a #[mysql(rename = "...")] override.
(get_group_trait_info): New function.
(pool): New function. mysql::Pool::new eagerly opens its default 10
connections before returning (about 7s against this database); every
query function here used to call it per request, paying that cost on
every single API call. Builds the Pool once behind a
OnceLock<Mutex<Option<Pool>>> and shares the cheap, Arc-backed handle
from then on (OnceLock::get_or_init can't be used alone since
Pool::new is fallible and get_or_try_init is still unstable).
(get_all_species, get_species_by_name, get_all_groups,
get_group_by_name, get_group_strains, get_group_trait_values,
get_group_trait_info): Use it instead of building their own Pool.
* src/lib.rs (GroupTraitLookup): Generalize over the found value (was
Vec<db::TraitValue>-specific), so group_trait_info can share it with
group_trait_values.
(get_group_trait_info_result): New function.
(group_trait_info): New handler for GET /groups/{group}/traits/{trait_id}.
(app): Register it.
* tests/common/mod.rs: New file. get() issues a GET against
genenetwork4::app() and warns on stderr if it takes over a second --
this is what surfaced the Pool::new cost above.
* tests/species.rs, tests/species_by_name.rs, tests/groups.rs,
tests/group_by_name.rs, tests/group_strains.rs,
tests/group_trait_values.rs: Use tests/common::get instead of building
each request by hand.
* tests/group_trait_info.rs, test/python/test_group_trait_info.py: New
files.
* tests/openapi.rs, test/python/test_openapi.py: Check the new path is
in the generated spec.
* README.md (GET /groups/{group}/traits/{trait_id}): Document it.
(Tests): List the new test files; document tests/common::get's timing
warnings and how to see them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
The old skip check connected to 127.0.0.1:3306, so tests ran (and
failed with 500) whenever a server was listening but the configured
credentials were rejected. tests/common/mod.rs now attempts a real
connection with the configured SQL_URI; tests skip only when the
database is genuinely unusable, and run for real otherwise.
|