about summary refs log tree commit diff
path: root/docs/dev
diff options
context:
space:
mode:
authorpjotr <pjotr.public01@thebird.nl>2026-11-09 12:16:10 +0000
committerpjotr <pjotr.public01@thebird.nl>2026-11-09 12:16:10 +0000
commit29eb1363caea8c0eb13eb91c5a44d4f2967b69da (patch)
tree12376f6e593653c1f078bb933c11c36da64471a9 /docs/dev
parent50fe1b7c0ef0175ce3a830700d78ebfed1c36f91 (diff)
downloadgenenetwork4-29eb1363caea8c0eb13eb91c5a44d4f2967b69da.tar.gz
move dev SKILL.md
Diffstat (limited to 'docs/dev')
-rw-r--r--docs/dev/SKILL.md349
1 files changed, 349 insertions, 0 deletions
diff --git a/docs/dev/SKILL.md b/docs/dev/SKILL.md
new file mode 100644
index 0000000..ce936d7
--- /dev/null
+++ b/docs/dev/SKILL.md
@@ -0,0 +1,349 @@
+---
+name: rust-axum-mariadb-api
+description: Build a Rust/Axum REST API with OpenAPI docs, RFC-7807 errors, and a read-only MariaDB backend, packaged with Guix
+---
+
+# Rust/Axum REST API over MariaDB
+
+Practices used to build genenetwork4. Reuse these when starting a
+similar Rust/Axum service backed by a MariaDB/MySQL database.
+
+## mysql crate
+
+- Pin `mysql = { version = "...", default-features = false, features =
+  ["minimal-rust", "derive"] }`. The default features pull in
+  `flate2`'s system-zlib backend, which fails to build in sandboxes
+  without a system zlib; `minimal-rust` uses a pure-Rust zlib instead.
+  `derive` is needed for `#[derive(FromRow)]`.
+- The client is synchronous. Run every query inside
+  `tokio::task::spawn_blocking` so it doesn't stall the async runtime.
+- Tuple-based `FromRow` (`conn.exec_first::<(A, B, ...), _, _>`) is
+  only implemented up to 12 elements. For a wider row, define a struct
+  and `#[derive(FromRow)]` instead.
+- `FromRow` derive matches columns by PascalCase of the field name by
+  default (`#[mysql(rename_all = "PascalCase")]` on the struct). Add
+  `#[mysql(rename = "ActualColumn")]` on a field whenever its Rust name
+  doesn't PascalCase back to the real column -- a lowercase column
+  (e.g. `public`), or a field deliberately renamed away from a
+  database-specific name (see "Naming" below).
+- Keep `mysql::params!` queries (`:name` placeholders) over string
+  interpolation even for read-only queries.
+- A query that joins several tables and selects same-named columns
+  from more than one (`Id`, `Units`, `Year`, ...) needs an explicit SQL
+  `AS` alias per column to disambiguate for `FromRow`. Rather than
+  reaching for `#[mysql(rename = "...")]` on every such field, write
+  each alias to already equal that field's default PascalCase
+  conversion (e.g. a `pubmed_id` field needs the column aliased
+  `AS PubmedId`, not `AS PubMedId` -- the derive capitalizes each
+  underscore-separated segment, it doesn't preserve mid-word
+  capitalization) -- then no renames are needed at all. Verify empirically
+  with a real query against the live database rather than trusting the
+  conversion by inspection; it is not compile-checked.
+- **`mysql::Pool::new` is expensive -- do not call it per request.** Its
+  default `PoolConstraints` (`min: 10, max: 100`) makes it eagerly open
+  10 real connections before returning, which took about 7 seconds
+  against a real database in this project despite running on
+  localhost (a single already-established connection then takes under
+  200ms per query) -- plausibly a per-connection handshake/auth cost
+  multiplied by 10, not network latency (a raw `TcpStream::connect` to
+  the same address was sub-millisecond). A handler that builds a fresh
+  `Pool` and drops it at the end of the function -- an easy mistake,
+  since `Pool::new` reads just like opening any other resource --
+  pays this cost on *every single request*. Build one `Pool` lazily on
+  first use and share it (it's a cheap, thread-safe, `Arc`-backed
+  handle, so cloning it out of a cache is free): `OnceLock<Mutex<Option<Pool>>>`
+  works, since `Pool::new` is fallible and `OnceLock::get_or_init`'s
+  closure isn't allowed to fail (`get_or_try_init` is still unstable,
+  rust-lang/rust#109737) -- lock, check, build-and-cache on a miss,
+  return a clone either way. This class of bug is exactly what
+  per-request query timing (see Testing below) is for: it's invisible
+  in a quick manual check (the person testing doesn't notice ~7s
+  consistently, they just think the service is slow) but immediately
+  obvious once requests are timed individually.
+
+## Naming: database vs. code
+
+When a table/column name is domain jargon, internal, or otherwise not
+the name the API should expose (e.g. `InbredSet` where GeneNetwork's
+own code and docs call the concept "Group"), keep the raw name ONLY in:
+
+- SQL query strings (`"... FROM InbredSet WHERE ..."`)
+- `#[mysql(rename = "...")]` values
+
+Everywhere else -- struct names, field names, route paths, doc prose
+about the Rust API -- use the human-facing alias. This keeps the
+MariaDB schema as the single place the historical name has to appear.
+
+## Error handling
+
+One shared RFC 7807 (Problem Details)-style error type for every
+endpoint, so clients parse one shape everywhere:
+
+- Fields beyond the RFC: `code` (machine-readable, stable), `detail`,
+  `operation_id` (matches the OpenAPI operation name), `spec_url`
+  (always the OpenAPI document's path, so a client can always look up
+  what it just failed to understand), `request_id`, and an optional
+  `next` hint pointing at a related endpoint (e.g. a 404 on
+  `/thing/{name}` sets `next` to `/thing`, so a caller with a typo can
+  discover valid values).
+- `type`/`spec_url` are relative paths, not absolute URLs, when the
+  service has no fixed public domain -- RFC 7807 allows `type` to be a
+  relative URI reference.
+- Document the same error schema on every route, including ones with
+  no failure path today (no DB access, no I/O) -- a handler gaining
+  one later doesn't need a doc update, and the spec stays honest about
+  the shape every endpoint *could* return.
+- Generate a request id per request (`tower_http::request_id`:
+  `SetRequestIdLayer` + `PropagateRequestIdLayer`, `x-request-id`) and
+  include it in every error body and the response headers, so a report
+  can be correlated with logs. Include the server's time so far
+  (`took_ms`) too; it is an RFC 7807 extension member, and it matches
+  the `meta.took_ms` of a success (see "Success responses").
+- Errors raised by an extractor (the 401 for a bad token) have no
+  handler to stamp them. Have the first extractor (`Ctx`) store the
+  request's id and start time in the request extensions, and have the
+  token extractor read them for its rejection; otherwise those errors
+  silently lack a `request_id`.
+
+## Success responses
+
+The success counterpart of the error type: one envelope for every
+endpoint, so clients (and LLM callers) parse one shape and are told
+what to do next. Do this from the start; retrofitting it later breaks
+every client and test at once.
+
+- Shape: `{ "data": ..., "meta": {"state", "took_ms", "request_id"},
+  "notes": [...], "next": [{"href", "description"}] }`. `data` is the
+  payload and nothing else.
+- `meta.state` is `complete` or `partial`, not a copy of the HTTP
+  status. Use `partial` (HTTP still 200) when a response is usable but
+  something it could have had is missing or unavailable -- an optional
+  lookup failed, results were left out for authorization, a ranking
+  covers only a window -- and say what in `notes`. The alternative,
+  quietly returning nulls, is invisible to callers.
+- `took_ms` is server time (it cannot be the client's round trip).
+  Measure it with an extractor that is the handler's first parameter
+  (`Ctx`: it records the start time and the request id), and build the
+  response with `ctx.envelope(data)` at the end, so no handler fills in
+  timing by hand and none forgets it.
+- `notes` are server-authored only. Never copy database text or request
+  input into them: LLM callers read them as instructions.
+- `next` hrefs are relative, percent-encoded (a name with a space is
+  `a%20b`), and must resolve: write a test that fetches an
+  endpoint of each kind and follows every `next` it suggests (skip the
+  huge listings, and say so in the suggestion's description). Suggest
+  concrete calls (the record's children, its related records, the next
+  page), not templates.
+- Exempt a plain-text liveness route and the OpenAPI document itself;
+  nothing else. Enforce it: a test over `/openapi.json` that fails if
+  any JSON 200 response is not an `Envelope_*` schema, so a new
+  endpoint cannot forget it.
+- In `#[utoipa::path]`, `body = Envelope<Vec<db::Thing>>`; the generic
+  `Envelope<T>` derives `ToSchema`.
+- In tests, parse every success body through one helper that checks the
+  envelope and returns `data`, rather than indexing `["data"]` by hand
+  at each site.
+
+## OpenAPI
+
+Use `utoipa` + `utoipa-axum`, not a hand-maintained spec file:
+
+- `#[utoipa::path(...)]` on each handler, `#[derive(utoipa::ToSchema)]`
+  on each request/response struct.
+- Register routes with `utoipa_axum::routes!(handler)` on an
+  `OpenApiRouter`, not plain `axum::routing::get`.
+- A `#[derive(utoipa::OpenApi)]` struct supplies `info` (title,
+  description, license) -- without it the spec carries utoipa-axum's
+  placeholder metadata.
+- `OpenApiRouter::split_for_parts()` yields the plain `axum::Router`
+  plus the assembled `OpenApi` value; serve the latter with
+  `Json` at `/openapi.json`.
+- This way a route or schema change that doesn't update its
+  `#[utoipa::path]`/`ToSchema` is a compile error or an obviously
+  incomplete diff, not spec drift noticed weeks later.
+- When most endpoints require the same thing (a bearer token, an API
+  key), set it once as the document default -- `#[openapi(security(
+  ("scheme_name" = [])), ...)]` on the `#[derive(utoipa::OpenApi)]`
+  struct -- instead of repeating `security(...)` on every
+  `#[utoipa::path]`. The handful of endpoints that must NOT require it
+  (health check, version, the endpoint that hands out the credential in
+  the first place) override with an empty `security()` of their own;
+  an absent `security(...)` on an operation inherits the document
+  default, a present-but-empty one overrides to "none" -- standard
+  OpenAPI semantics, not utoipa-specific. Audit every operation's
+  resolved `security` in the generated `/openapi.json` after making this
+  change (`"security": [...]` present vs. absent per operation) -- it's
+  easy to silently document a public endpoint as requiring auth, or
+  vice versa, since nothing fails to compile either way.
+- A response shape repeated across many handlers (e.g. "401, missing or
+  invalid token", always the same error body type) doesn't need
+  hand-copying into every `responses(...)` list: give it its own
+  `#[derive(utoipa::IntoResponses)] #[response(status = 401, ...)]`
+  marker struct wrapping the real body type (`struct Unauthorized(
+  #[to_schema] ApiError)`), then reference it bare inside any handler's
+  `responses(...)` (`responses(Unauthorized, (status = 200, ...))`). The
+  marker struct is never constructed at runtime (the actual response
+  still comes from wherever the rejection/error is produced, e.g. an
+  extractor's `FromRequestParts` impl) -- `#[allow(dead_code)]` it with
+  a comment saying why, rather than leaving an unexplained warning or
+  actually constructing a throwaway value just to silence one.
+
+## Authentication/authorization
+
+- Make requiring a token **opt-in per handler**, not a blanket
+  middleware layer: a custom extractor (`struct AuthToken; impl
+  FromRequestParts<S> for AuthToken`) that verifies/rejects in
+  `from_request_parts`. A handler adds `token: AuthToken` (or `_token:
+  AuthToken` if it only needs the gate, not the claims) to its
+  parameter list to require one; omitting the parameter is what makes
+  it optional, with no central route-list to keep in sync. Non-body
+  extractors like this can go anywhere in the parameter list except
+  after a body-consuming one (`Json<T>` etc. must stay last).
+- If some data is "public" and some isn't, don't let that become "some
+  endpoints need a token and some don't" -- keep the token check
+  universal (cheap: local signature verification, no network call) and
+  make "is this specific resource public" a second, independent
+  decision the handler makes afterward, with its own cache, only
+  calling out to an authorization service for the resources that
+  actually need it.
+- A resource addressed by `(container, item)` where `container` can
+  belong to more than one underlying category (e.g. a "dataset" that's
+  sometimes one database table, sometimes another) should resolve
+  `container`'s category **first**, as its own lookup, before ever
+  looking up `item`. Resolving `item` by guessing across categories
+  (try category A's lookup, fall back to category B's) risks a same-named
+  collision silently returning the wrong thing -- resolving `container`
+  first instead means only a much smaller set of names (containers) is
+  ever disambiguated across categories, not every item name ever.
+- An endpoint that hands out a credential (e.g. proxies an auth
+  service's own token grant for a well-known test user) should decide
+  per request, from the connection, whether to answer -- not from a mode
+  flag or a bind address, which are easy to leave set wrongly in a
+  deployment. Answer only a caller whose TCP peer address is loopback
+  (`axum::extract::ConnectInfo`; serve with
+  `into_make_service_with_connect_info`), and fail closed when there is
+  no peer address. Never establish "local" from a header, but do refuse a
+  request that carries `X-Forwarded-For`/`Forwarded`/`X-Real-IP`: a
+  reverse proxy on the same host connects from loopback for every client,
+  so without that refusal the whole internet looks local. Keep the
+  decision in small plain functions with their own unit tests, test the
+  handler with injected peer addresses (a local one, several remote
+  ones, none at all, and a local one with a proxy header), and list the
+  403 in the OpenAPI responses. Default any credentials the endpoint
+  needs to the target service's own published "dev fixtures" (a
+  well-known test client/user every checkout of that service already
+  has) so it works with zero extra configuration.
+  Also give deployments a way to leave the route out entirely (a config
+  switch checked when the router is built, so the route is a 404 and
+  absent from the OpenAPI spec). Make that switch fail closed: parse it
+  strictly so only a clearly affirmative value keeps the endpoint on and
+  a typo ("flase") turns it off, since the safe state of a
+  credential-issuing endpoint is "not running".
+  service already has) so it works with zero extra configuration.
+
+## Testing
+
+Two parallel suites exercising the same endpoints:
+
+- Rust integration tests (`tests/*.rs`) call the router in-process via
+  `tower::ServiceExt::oneshot(app())` -- no socket needed.
+- Python tests (`test/python/test_*.py`) spawn the actual compiled
+  binary and talk to it over real HTTP, since that's the only way
+  Python can reach it; share a `ServerTestCase` base class that spawns
+  the binary and polls for the port.
+- Any test that needs the real database checks reachability first
+  (`TcpStream::connect_timeout` / a `socket.create_connection` helper)
+  and skips with a message rather than failing when unreachable --
+  the Guix build sandbox has no network access, and skip (not failure)
+  is the correct result there.
+- Use one well-known, always-present fixture row per table (e.g. mouse
+  and rat for a species table, BXD for a groups table) rather than
+  inventing synthetic data -- grep the existing Python test suites of
+  sibling services for what they already rely on being present.
+- Route every DB-backed Rust test's request through one shared helper
+  (e.g. `tests/common::get`, a `mod common;` under `tests/` so every
+  integration-test file can use it) that times the call and warns on
+  stderr past a threshold (e.g. 1s). This is what actually caught the
+  `Pool::new`-per-request bug above -- it was invisible until requests
+  were timed individually. Warnings only show with `cargo test --
+  --nocapture` (or on a failing test; cargo captures output for
+  passing tests otherwise), so say so in the README. Each integration
+  test file compiles to its own process, so the one-time cost of
+  building a share-once resource (like the Pool above) still shows up
+  once per test *file*, not just once overall -- that's expected, not
+  a sign the fix didn't work; what matters is that repeated calls
+  inside one file (or one real server process) stay fast after it.
+- If endpoints need a bearer token and the service has a local-only
+  token-issuing endpoint (see "Authentication/authorization" above),
+  have the test suite fetch its own token from it rather than requiring
+  a human to mint and export one by hand first. Try that first, fall
+  back to an externally-provided token (an env var) when there is none
+  -- so the suite still works unmodified either way. For the in-process
+  Rust suite this means calling the router's own endpoint via the same
+  `oneshot` plumbing used for every other request, with a local peer
+  address injected into the request (there is no real connection);
+  for a suite that spawns the real binary as a subprocess (Python,
+  here), the test client is already local, so the subprocess answers
+  its own token request over the same HTTP connection the test already
+  uses.
+- Mutating process-wide state (env vars, global config) from a test is
+  only safe when nothing else in the same process can read or write
+  that same state concurrently. A single compiled test binary (one
+  `tests/*.rs` file) runs its `#[tokio::test]`/`#[test]` functions
+  concurrently on different threads by default -- so a test that must
+  flip an env var through several values keeps all of those mutations
+  in one sequential test function, with a comment saying why, rather
+  than splitting them across multiple test functions in the same file.
+  A suite that instead spawns a fresh subprocess per test class (own
+  process, own env, e.g. Python's `ServerTestCase`) doesn't have this
+  problem -- each class can set its own env independently and safely.
+  On edition 2024, `std::env::set_var`/`remove_var` are `unsafe fn`
+  (exactly because of this hazard); wrap each call in its own tiny
+  `unsafe { ... }` with a `// SAFETY:` comment pointing at whichever of
+  the two guarantees above makes it sound here, rather than one broad
+  `unsafe` block around unrelated code.
+
+## Guix packaging
+
+- `guix import crate -f Cargo.lock PACKAGE` regenerates the crate
+  dependency list; diff the crate-name list before/after with `comm
+  -23` (both directions) to confirm no unrelated version silently
+  drifted.
+- `guix shell -C -D -N -f guix.scm` (isolated + network) can silently
+  drop build dependencies on some Guix versions when `-D` and `-N` are
+  combined -- if a fully isolated shell reports "cargo: command not
+  found", fall back to `guix shell -D -f guix.scm` (no isolation).
+- Known trap: a committed `Cargo.lock` can pin crate versions newer
+  than the Guix-packaged rust supports (crates.io keeps publishing;
+  cargo's own relocking step inside the build will refuse with
+  "rustc X.Y is not supported by CRATE, requires rustc Z"). This is a
+  packaging-environment mismatch, not a code bug -- rule that out
+  before assuming a build failure is caused by whatever code change
+  prompted the build. To fix it:
+  - `cargo update -p CRATE --precise VERSION`, one crate at a time (per
+    the error list), picks the newest release still compatible --
+    `cargo generate-lockfile` alone re-derives these correctly for
+    most transitive deps (it already favors the rustc-compatible
+    version), but re-running it also undoes any manual `--precise` pin
+    you already made, so apply pins one at a time without regenerating
+    the whole lockfile afterwards.
+  - A crate's own non-macro code can need a newer rustc regardless of
+    which version you pick -- e.g. mysql_common adopted `let`-chains
+    and `is_multiple_of` (both stabilized in rustc 1.88) in every
+    0.36.x/0.37.x release, so no `--precise` pin within those lines
+    helps; the fix was downgrading the *dependent* (`mysql` in
+    Cargo.toml) to the last major version pinning the still-compatible
+    mysql_common 0.35.x. Check a candidate version actually builds,
+    don't assume a lower semver number is automatically safe.
+  - Query `https://index.crates.io/<2-or-3-letter-prefix-dirs>/<name>`
+    (the sparse registry index, one JSON line per version, with a
+    `rust_version` field when the crate declares an MSRV) to compare
+    versions without a full `cargo update` round-trip per guess.
+
+## Commits
+
+Force the author explicitly
+(`--author="Name <email>"`) if the local git identity doesn't match,
+and write messages in the GNU ChangeLog style: `* path/file (entity):
+what changed.`, one bullet per file/entity, most specific context
+first.