diff options
| author | pjotr <pjotr.public01@thebird.nl> | 2026-11-09 12:16:10 +0000 |
|---|---|---|
| committer | pjotr <pjotr.public01@thebird.nl> | 2026-11-09 12:16:10 +0000 |
| commit | 29eb1363caea8c0eb13eb91c5a44d4f2967b69da (patch) | |
| tree | 12376f6e593653c1f078bb933c11c36da64471a9 /docs/dev | |
| parent | 50fe1b7c0ef0175ce3a830700d78ebfed1c36f91 (diff) | |
| download | genenetwork4-29eb1363caea8c0eb13eb91c5a44d4f2967b69da.tar.gz | |
move dev SKILL.md
Diffstat (limited to 'docs/dev')
| -rw-r--r-- | docs/dev/SKILL.md | 349 |
1 files changed, 349 insertions, 0 deletions
diff --git a/docs/dev/SKILL.md b/docs/dev/SKILL.md new file mode 100644 index 0000000..ce936d7 --- /dev/null +++ b/docs/dev/SKILL.md @@ -0,0 +1,349 @@ +--- +name: rust-axum-mariadb-api +description: Build a Rust/Axum REST API with OpenAPI docs, RFC-7807 errors, and a read-only MariaDB backend, packaged with Guix +--- + +# Rust/Axum REST API over MariaDB + +Practices used to build genenetwork4. Reuse these when starting a +similar Rust/Axum service backed by a MariaDB/MySQL database. + +## mysql crate + +- Pin `mysql = { version = "...", default-features = false, features = + ["minimal-rust", "derive"] }`. The default features pull in + `flate2`'s system-zlib backend, which fails to build in sandboxes + without a system zlib; `minimal-rust` uses a pure-Rust zlib instead. + `derive` is needed for `#[derive(FromRow)]`. +- The client is synchronous. Run every query inside + `tokio::task::spawn_blocking` so it doesn't stall the async runtime. +- Tuple-based `FromRow` (`conn.exec_first::<(A, B, ...), _, _>`) is + only implemented up to 12 elements. For a wider row, define a struct + and `#[derive(FromRow)]` instead. +- `FromRow` derive matches columns by PascalCase of the field name by + default (`#[mysql(rename_all = "PascalCase")]` on the struct). Add + `#[mysql(rename = "ActualColumn")]` on a field whenever its Rust name + doesn't PascalCase back to the real column -- a lowercase column + (e.g. `public`), or a field deliberately renamed away from a + database-specific name (see "Naming" below). +- Keep `mysql::params!` queries (`:name` placeholders) over string + interpolation even for read-only queries. +- A query that joins several tables and selects same-named columns + from more than one (`Id`, `Units`, `Year`, ...) needs an explicit SQL + `AS` alias per column to disambiguate for `FromRow`. Rather than + reaching for `#[mysql(rename = "...")]` on every such field, write + each alias to already equal that field's default PascalCase + conversion (e.g. a `pubmed_id` field needs the column aliased + `AS PubmedId`, not `AS PubMedId` -- the derive capitalizes each + underscore-separated segment, it doesn't preserve mid-word + capitalization) -- then no renames are needed at all. Verify empirically + with a real query against the live database rather than trusting the + conversion by inspection; it is not compile-checked. +- **`mysql::Pool::new` is expensive -- do not call it per request.** Its + default `PoolConstraints` (`min: 10, max: 100`) makes it eagerly open + 10 real connections before returning, which took about 7 seconds + against a real database in this project despite running on + localhost (a single already-established connection then takes under + 200ms per query) -- plausibly a per-connection handshake/auth cost + multiplied by 10, not network latency (a raw `TcpStream::connect` to + the same address was sub-millisecond). A handler that builds a fresh + `Pool` and drops it at the end of the function -- an easy mistake, + since `Pool::new` reads just like opening any other resource -- + pays this cost on *every single request*. Build one `Pool` lazily on + first use and share it (it's a cheap, thread-safe, `Arc`-backed + handle, so cloning it out of a cache is free): `OnceLock<Mutex<Option<Pool>>>` + works, since `Pool::new` is fallible and `OnceLock::get_or_init`'s + closure isn't allowed to fail (`get_or_try_init` is still unstable, + rust-lang/rust#109737) -- lock, check, build-and-cache on a miss, + return a clone either way. This class of bug is exactly what + per-request query timing (see Testing below) is for: it's invisible + in a quick manual check (the person testing doesn't notice ~7s + consistently, they just think the service is slow) but immediately + obvious once requests are timed individually. + +## Naming: database vs. code + +When a table/column name is domain jargon, internal, or otherwise not +the name the API should expose (e.g. `InbredSet` where GeneNetwork's +own code and docs call the concept "Group"), keep the raw name ONLY in: + +- SQL query strings (`"... FROM InbredSet WHERE ..."`) +- `#[mysql(rename = "...")]` values + +Everywhere else -- struct names, field names, route paths, doc prose +about the Rust API -- use the human-facing alias. This keeps the +MariaDB schema as the single place the historical name has to appear. + +## Error handling + +One shared RFC 7807 (Problem Details)-style error type for every +endpoint, so clients parse one shape everywhere: + +- Fields beyond the RFC: `code` (machine-readable, stable), `detail`, + `operation_id` (matches the OpenAPI operation name), `spec_url` + (always the OpenAPI document's path, so a client can always look up + what it just failed to understand), `request_id`, and an optional + `next` hint pointing at a related endpoint (e.g. a 404 on + `/thing/{name}` sets `next` to `/thing`, so a caller with a typo can + discover valid values). +- `type`/`spec_url` are relative paths, not absolute URLs, when the + service has no fixed public domain -- RFC 7807 allows `type` to be a + relative URI reference. +- Document the same error schema on every route, including ones with + no failure path today (no DB access, no I/O) -- a handler gaining + one later doesn't need a doc update, and the spec stays honest about + the shape every endpoint *could* return. +- Generate a request id per request (`tower_http::request_id`: + `SetRequestIdLayer` + `PropagateRequestIdLayer`, `x-request-id`) and + include it in every error body and the response headers, so a report + can be correlated with logs. Include the server's time so far + (`took_ms`) too; it is an RFC 7807 extension member, and it matches + the `meta.took_ms` of a success (see "Success responses"). +- Errors raised by an extractor (the 401 for a bad token) have no + handler to stamp them. Have the first extractor (`Ctx`) store the + request's id and start time in the request extensions, and have the + token extractor read them for its rejection; otherwise those errors + silently lack a `request_id`. + +## Success responses + +The success counterpart of the error type: one envelope for every +endpoint, so clients (and LLM callers) parse one shape and are told +what to do next. Do this from the start; retrofitting it later breaks +every client and test at once. + +- Shape: `{ "data": ..., "meta": {"state", "took_ms", "request_id"}, + "notes": [...], "next": [{"href", "description"}] }`. `data` is the + payload and nothing else. +- `meta.state` is `complete` or `partial`, not a copy of the HTTP + status. Use `partial` (HTTP still 200) when a response is usable but + something it could have had is missing or unavailable -- an optional + lookup failed, results were left out for authorization, a ranking + covers only a window -- and say what in `notes`. The alternative, + quietly returning nulls, is invisible to callers. +- `took_ms` is server time (it cannot be the client's round trip). + Measure it with an extractor that is the handler's first parameter + (`Ctx`: it records the start time and the request id), and build the + response with `ctx.envelope(data)` at the end, so no handler fills in + timing by hand and none forgets it. +- `notes` are server-authored only. Never copy database text or request + input into them: LLM callers read them as instructions. +- `next` hrefs are relative, percent-encoded (a name with a space is + `a%20b`), and must resolve: write a test that fetches an + endpoint of each kind and follows every `next` it suggests (skip the + huge listings, and say so in the suggestion's description). Suggest + concrete calls (the record's children, its related records, the next + page), not templates. +- Exempt a plain-text liveness route and the OpenAPI document itself; + nothing else. Enforce it: a test over `/openapi.json` that fails if + any JSON 200 response is not an `Envelope_*` schema, so a new + endpoint cannot forget it. +- In `#[utoipa::path]`, `body = Envelope<Vec<db::Thing>>`; the generic + `Envelope<T>` derives `ToSchema`. +- In tests, parse every success body through one helper that checks the + envelope and returns `data`, rather than indexing `["data"]` by hand + at each site. + +## OpenAPI + +Use `utoipa` + `utoipa-axum`, not a hand-maintained spec file: + +- `#[utoipa::path(...)]` on each handler, `#[derive(utoipa::ToSchema)]` + on each request/response struct. +- Register routes with `utoipa_axum::routes!(handler)` on an + `OpenApiRouter`, not plain `axum::routing::get`. +- A `#[derive(utoipa::OpenApi)]` struct supplies `info` (title, + description, license) -- without it the spec carries utoipa-axum's + placeholder metadata. +- `OpenApiRouter::split_for_parts()` yields the plain `axum::Router` + plus the assembled `OpenApi` value; serve the latter with + `Json` at `/openapi.json`. +- This way a route or schema change that doesn't update its + `#[utoipa::path]`/`ToSchema` is a compile error or an obviously + incomplete diff, not spec drift noticed weeks later. +- When most endpoints require the same thing (a bearer token, an API + key), set it once as the document default -- `#[openapi(security( + ("scheme_name" = [])), ...)]` on the `#[derive(utoipa::OpenApi)]` + struct -- instead of repeating `security(...)` on every + `#[utoipa::path]`. The handful of endpoints that must NOT require it + (health check, version, the endpoint that hands out the credential in + the first place) override with an empty `security()` of their own; + an absent `security(...)` on an operation inherits the document + default, a present-but-empty one overrides to "none" -- standard + OpenAPI semantics, not utoipa-specific. Audit every operation's + resolved `security` in the generated `/openapi.json` after making this + change (`"security": [...]` present vs. absent per operation) -- it's + easy to silently document a public endpoint as requiring auth, or + vice versa, since nothing fails to compile either way. +- A response shape repeated across many handlers (e.g. "401, missing or + invalid token", always the same error body type) doesn't need + hand-copying into every `responses(...)` list: give it its own + `#[derive(utoipa::IntoResponses)] #[response(status = 401, ...)]` + marker struct wrapping the real body type (`struct Unauthorized( + #[to_schema] ApiError)`), then reference it bare inside any handler's + `responses(...)` (`responses(Unauthorized, (status = 200, ...))`). The + marker struct is never constructed at runtime (the actual response + still comes from wherever the rejection/error is produced, e.g. an + extractor's `FromRequestParts` impl) -- `#[allow(dead_code)]` it with + a comment saying why, rather than leaving an unexplained warning or + actually constructing a throwaway value just to silence one. + +## Authentication/authorization + +- Make requiring a token **opt-in per handler**, not a blanket + middleware layer: a custom extractor (`struct AuthToken; impl + FromRequestParts<S> for AuthToken`) that verifies/rejects in + `from_request_parts`. A handler adds `token: AuthToken` (or `_token: + AuthToken` if it only needs the gate, not the claims) to its + parameter list to require one; omitting the parameter is what makes + it optional, with no central route-list to keep in sync. Non-body + extractors like this can go anywhere in the parameter list except + after a body-consuming one (`Json<T>` etc. must stay last). +- If some data is "public" and some isn't, don't let that become "some + endpoints need a token and some don't" -- keep the token check + universal (cheap: local signature verification, no network call) and + make "is this specific resource public" a second, independent + decision the handler makes afterward, with its own cache, only + calling out to an authorization service for the resources that + actually need it. +- A resource addressed by `(container, item)` where `container` can + belong to more than one underlying category (e.g. a "dataset" that's + sometimes one database table, sometimes another) should resolve + `container`'s category **first**, as its own lookup, before ever + looking up `item`. Resolving `item` by guessing across categories + (try category A's lookup, fall back to category B's) risks a same-named + collision silently returning the wrong thing -- resolving `container` + first instead means only a much smaller set of names (containers) is + ever disambiguated across categories, not every item name ever. +- An endpoint that hands out a credential (e.g. proxies an auth + service's own token grant for a well-known test user) should decide + per request, from the connection, whether to answer -- not from a mode + flag or a bind address, which are easy to leave set wrongly in a + deployment. Answer only a caller whose TCP peer address is loopback + (`axum::extract::ConnectInfo`; serve with + `into_make_service_with_connect_info`), and fail closed when there is + no peer address. Never establish "local" from a header, but do refuse a + request that carries `X-Forwarded-For`/`Forwarded`/`X-Real-IP`: a + reverse proxy on the same host connects from loopback for every client, + so without that refusal the whole internet looks local. Keep the + decision in small plain functions with their own unit tests, test the + handler with injected peer addresses (a local one, several remote + ones, none at all, and a local one with a proxy header), and list the + 403 in the OpenAPI responses. Default any credentials the endpoint + needs to the target service's own published "dev fixtures" (a + well-known test client/user every checkout of that service already + has) so it works with zero extra configuration. + Also give deployments a way to leave the route out entirely (a config + switch checked when the router is built, so the route is a 404 and + absent from the OpenAPI spec). Make that switch fail closed: parse it + strictly so only a clearly affirmative value keeps the endpoint on and + a typo ("flase") turns it off, since the safe state of a + credential-issuing endpoint is "not running". + service already has) so it works with zero extra configuration. + +## Testing + +Two parallel suites exercising the same endpoints: + +- Rust integration tests (`tests/*.rs`) call the router in-process via + `tower::ServiceExt::oneshot(app())` -- no socket needed. +- Python tests (`test/python/test_*.py`) spawn the actual compiled + binary and talk to it over real HTTP, since that's the only way + Python can reach it; share a `ServerTestCase` base class that spawns + the binary and polls for the port. +- Any test that needs the real database checks reachability first + (`TcpStream::connect_timeout` / a `socket.create_connection` helper) + and skips with a message rather than failing when unreachable -- + the Guix build sandbox has no network access, and skip (not failure) + is the correct result there. +- Use one well-known, always-present fixture row per table (e.g. mouse + and rat for a species table, BXD for a groups table) rather than + inventing synthetic data -- grep the existing Python test suites of + sibling services for what they already rely on being present. +- Route every DB-backed Rust test's request through one shared helper + (e.g. `tests/common::get`, a `mod common;` under `tests/` so every + integration-test file can use it) that times the call and warns on + stderr past a threshold (e.g. 1s). This is what actually caught the + `Pool::new`-per-request bug above -- it was invisible until requests + were timed individually. Warnings only show with `cargo test -- + --nocapture` (or on a failing test; cargo captures output for + passing tests otherwise), so say so in the README. Each integration + test file compiles to its own process, so the one-time cost of + building a share-once resource (like the Pool above) still shows up + once per test *file*, not just once overall -- that's expected, not + a sign the fix didn't work; what matters is that repeated calls + inside one file (or one real server process) stay fast after it. +- If endpoints need a bearer token and the service has a local-only + token-issuing endpoint (see "Authentication/authorization" above), + have the test suite fetch its own token from it rather than requiring + a human to mint and export one by hand first. Try that first, fall + back to an externally-provided token (an env var) when there is none + -- so the suite still works unmodified either way. For the in-process + Rust suite this means calling the router's own endpoint via the same + `oneshot` plumbing used for every other request, with a local peer + address injected into the request (there is no real connection); + for a suite that spawns the real binary as a subprocess (Python, + here), the test client is already local, so the subprocess answers + its own token request over the same HTTP connection the test already + uses. +- Mutating process-wide state (env vars, global config) from a test is + only safe when nothing else in the same process can read or write + that same state concurrently. A single compiled test binary (one + `tests/*.rs` file) runs its `#[tokio::test]`/`#[test]` functions + concurrently on different threads by default -- so a test that must + flip an env var through several values keeps all of those mutations + in one sequential test function, with a comment saying why, rather + than splitting them across multiple test functions in the same file. + A suite that instead spawns a fresh subprocess per test class (own + process, own env, e.g. Python's `ServerTestCase`) doesn't have this + problem -- each class can set its own env independently and safely. + On edition 2024, `std::env::set_var`/`remove_var` are `unsafe fn` + (exactly because of this hazard); wrap each call in its own tiny + `unsafe { ... }` with a `// SAFETY:` comment pointing at whichever of + the two guarantees above makes it sound here, rather than one broad + `unsafe` block around unrelated code. + +## Guix packaging + +- `guix import crate -f Cargo.lock PACKAGE` regenerates the crate + dependency list; diff the crate-name list before/after with `comm + -23` (both directions) to confirm no unrelated version silently + drifted. +- `guix shell -C -D -N -f guix.scm` (isolated + network) can silently + drop build dependencies on some Guix versions when `-D` and `-N` are + combined -- if a fully isolated shell reports "cargo: command not + found", fall back to `guix shell -D -f guix.scm` (no isolation). +- Known trap: a committed `Cargo.lock` can pin crate versions newer + than the Guix-packaged rust supports (crates.io keeps publishing; + cargo's own relocking step inside the build will refuse with + "rustc X.Y is not supported by CRATE, requires rustc Z"). This is a + packaging-environment mismatch, not a code bug -- rule that out + before assuming a build failure is caused by whatever code change + prompted the build. To fix it: + - `cargo update -p CRATE --precise VERSION`, one crate at a time (per + the error list), picks the newest release still compatible -- + `cargo generate-lockfile` alone re-derives these correctly for + most transitive deps (it already favors the rustc-compatible + version), but re-running it also undoes any manual `--precise` pin + you already made, so apply pins one at a time without regenerating + the whole lockfile afterwards. + - A crate's own non-macro code can need a newer rustc regardless of + which version you pick -- e.g. mysql_common adopted `let`-chains + and `is_multiple_of` (both stabilized in rustc 1.88) in every + 0.36.x/0.37.x release, so no `--precise` pin within those lines + helps; the fix was downgrading the *dependent* (`mysql` in + Cargo.toml) to the last major version pinning the still-compatible + mysql_common 0.35.x. Check a candidate version actually builds, + don't assume a lower semver number is automatically safe. + - Query `https://index.crates.io/<2-or-3-letter-prefix-dirs>/<name>` + (the sparse registry index, one JSON line per version, with a + `rust_version` field when the crate declares an MSRV) to compare + versions without a full `cargo update` round-trip per guess. + +## Commits + +Force the author explicitly +(`--author="Name <email>"`) if the local git identity doesn't match, +and write messages in the GNU ChangeLog style: `* path/file (entity): +what changed.`, one bullet per file/entity, most specific context +first. |
