about summary refs log tree commit diff
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-08 13:28:15 +0000
committerPjotr <wrk@bosa.lan>2026-11-08 13:28:15 +0000
commite5c2aa0ab1e568fe91c63cd5257aee7ff08ebed9 (patch)
tree3404fdf0a0c1a0cd34218832905bac0361074b64
parentb8fdbb3d60603c07cacf9f406ed1a5c228223829 (diff)
downloadgenenetwork4-e5c2aa0ab1e568fe91c63cd5257aee7ff08ebed9.tar.gz
docs: what the last two commits changed, and the dev/production split
* README.md (Running with cargo): "This deployment runs two instances:
3000 (production) and 3001 (development)" -- the table of ports, binaries
and cache directories, both start-up invocations, and why the caches are
separate while the database, index and auth stub are shared. Also notes
that in-memory state is per instance, so the dev server is not a capacity
test of production.
* docs/www/ai-instruction.md:
  - /version now described honestly: git (with -dirty), dirty,
    source_digest, database reachable/unreachable -- and that
    `unreachable` means stop and report, not retry in a loop.
  - error table: 413 (plain text, not the JSON object: over the 64 MiB
    body limit), 500 database_error with a connection-loss detail is
    transient (retry once), and 409 run_id_in_use pointing at the status
    endpoint.
  - covariates: case_attribute is the smallest form (nothing posted),
    which is also the answer to a 413 from sending a value per individual.
-rw-r--r--README.md51
-rw-r--r--docs/www/ai-instruction.md27
2 files changed, 76 insertions, 2 deletions
diff --git a/README.md b/README.md
index dd3716b..a0f5958 100644
--- a/README.md
+++ b/README.md
@@ -1565,6 +1565,57 @@ GENENETWORK4_PORT=8080 cargo run
 GENENETWORK4_BIND=127.0.0.1 cargo run
 ```
 
+### This deployment runs two instances: 3000 (production) and 3001 (development)
+
+The reference deployment keeps a development instance next to the live
+one, on the same box, so work can be tried out without disturbing what
+the site is serving:
+
+| instance | port | binary | run caches | proxied by nginx |
+|---|---|---|---|---|
+| **production** | `3000` | `target/release/genenetwork4` | defaults (`/tmp/gn4-gemma`, `/tmp/gn4-rqtl`, …) | yes, at `/gn4/` (see [docs/nginx.md](docs/nginx.md)) |
+| **development** | `3001` | `target/debug/genenetwork4` | `/tmp/gn4-dev/{gemma,rqtl,ravanan}` | no |
+
+They differ only in the port, the build profile and those cache
+directories; both use the same read-only `SQL_URI`, the same Xapian index
+and the same auth stub, so a request behaves the same on either. The
+separate caches matter: a GEMMA or R/qtl run on the development instance
+must not fill, or be sped up by, production's warm cache.
+
+```sh
+P=$(guix shell -D -f guix.scm -- bash -c 'echo $GUIX_ENVIRONMENT')   # needs R + gemma-wrapper on PATH
+
+# production
+PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ \
+GENENETWORK4_BIND=127.0.0.1 GENENETWORK4_PORT=3000 GENENETWORK4_TEST_TOKEN=false \
+AUTH_SERVER_URL=http://127.0.0.1:8081/ \
+SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \
+XAPIAN_DB_PATH=/home/wrk/tmp/ai-root/factory/gn4-index SEARCH_ENGINE=native \
+nohup ./target/release/genenetwork4 > /tmp/gn4-3000.log 2>&1 &   # pid in /tmp/gn4.pid
+
+# development: same, plus the port and its own caches
+PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ \
+GENENETWORK4_BIND=127.0.0.1 GENENETWORK4_PORT=3001 GENENETWORK4_TEST_TOKEN=false \
+AUTH_SERVER_URL=http://127.0.0.1:8081/ \
+SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \
+XAPIAN_DB_PATH=/home/wrk/tmp/ai-root/factory/gn4-index SEARCH_ENGINE=native \
+GEMMA_CACHE_DIR=/tmp/gn4-dev/gemma RQTL_CACHE_DIR=/tmp/gn4-dev/rqtl \
+RAVANAN_CACHE_DIR=/tmp/gn4-dev/ravanan \
+nohup ./target/debug/genenetwork4 > /tmp/gn4-dev.log 2>&1 &   # pid in /tmp/gn4-dev.pid
+```
+
+Both need the `PATH`/`R_LIBS_SITE` pair (see "Building with cargo, outside
+a bare toolchain" below) or R/qtl fails with `there is no package called
+'optparse'`, and both need the auth stub on `127.0.0.1:8081` to serve
+`/auth/public-jwks` (see "Tests").
+
+Anything the server keeps in memory is per instance: the pool, the JWKS
+and authorisation caches, and any in-process compute limit. So the two
+instances do not serialise against each other — a mapping on `:3001` runs
+alongside one on `:3000` — which is the point of the split, and a good
+reason not to treat the development instance as a capacity test of what
+production will do.
+
 ### Building with cargo, outside a bare toolchain
 
 `cargo build` in the package's own dependency shell is the supported
diff --git a/docs/www/ai-instruction.md b/docs/www/ai-instruction.md
index d38e207..794167b 100644
--- a/docs/www/ai-instruction.md
+++ b/docs/www/ai-instruction.md
@@ -123,9 +123,19 @@ Two routes are safe to call first and need no token:
 
 ```sh
 curl -s "$API/openapi.json"   # every endpoint, parameter, schema
-curl -s "$API/version"        # build + git revision
+curl -s "$API/version"        # build, git revision, and whether the database answers
 ```
 
+`/version` is worth reading when something fails everywhere at once. It
+returns `git` (the commit the running binary was built from; a `-dirty`
+suffix means that build had uncommitted changes), `dirty`,
+`source_digest` (which identifies the build when a commit hash cannot),
+`database` (`reachable` or `unreachable`) and `date` (the *commit's* date,
+not the build time). If `database` is `unreachable`, every data endpoint
+will fail with `503 db_unavailable` or `500 database_error` and there is
+nothing you can do but report it — do not retry in a loop and do not
+conclude the resource does not exist.
+
 Every response is one envelope:
 
 | member | meaning |
@@ -409,7 +419,11 @@ A mapping's response ends with what to do next:
 
 **Covariates.** Add `covariates` to fit the study's own metadata — sex,
 diet, batch — alongside the trait. The easiest form names a case
-attribute (§3), and the server codes it:
+attribute (§3), and the server codes it. It is also the smallest: nothing
+is posted, where an inline covariate sends a value per individual — for a
+panel of thousands that is the difference between a few hundred bytes and
+a multi-megabyte body, and bodies are capped at 64 MiB (`413`, in plain
+text, if you pass it):
 
 ```json
 "covariates": [{"case_attribute": "Sex"}]
@@ -522,6 +536,7 @@ curl -s -H "$AUTH" "$API/groups/BXD/phewas?chr=9&mb=29.940961&window=10000"
 | `window` | **half-width in base pairs, centred**: `10000` is 10 kb up *and* downstream (the default) |
 | `genotype_version` | whose marker table defines the window; omit for the group's current one |
 
+
 ### What comes back
 
 `data` has `group`, `genotype_version`, `centre` (`locus`, `chr`, `mb`),
@@ -594,9 +609,17 @@ Errors are RFC 7807-style JSON (`type`, `title`, `status`, `code`,
 | 403 | `forbidden` | the data is not public: do not retry |
 | 404 | `not_found` | check the name; `next` often lists the valid values |
 | 422 | `invalid_query`, `invalid_lmm` | fix the query or parameter; do not retry |
+| 409 | `run_id_in_use` | the `?id=` you chose is already in use: pick another, or poll `/compute/{rqtl,gemma}/status/{id}` |
+| 413 | — (plain text, **not** JSON) | the request body is over the 64 MiB limit: send fewer values, or none at all (§4a's `case_attribute`) |
 | 429 | — | back off, honour `Retry-After` (not currently emitted, but written into the API's contract) |
+| 500 | `database_error` | if the `detail` mentions a connection (`Broken pipe`, `Connection reset`, `server has gone away`) it is transient: retry **once**, then report; otherwise treat it as a bug and report it with the `request_id` |
 | 502 / 503 | `backend_unreachable`, `db_unavailable` | transient: retry with exponential backoff, max ~5 tries |
 
+Two things about the error shape itself: a **413 comes back as plain
+text**, not the JSON error object (`Failed to buffer the request body:
+length limit exceeded`) — the body limit is enforced before any handler
+runs — and errors carry `took_ms` as well, like a success's `meta`.
+
 Quote `request_id` when reporting a problem — it is in the body, in the
 `x-request-id` header, and in the server's access log.