about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorpjotr <pjotr.public01@thebird.nl>2026-11-09 12:10:43 +0000
committerpjotr <pjotr.public01@thebird.nl>2026-11-09 12:10:43 +0000
commit50fe1b7c0ef0175ce3a830700d78ebfed1c36f91 (patch)
tree272c8e5204ac9d287ecdf7049d8f8ee386fb027a /docs
parenta4ee14ca455103fcad6cebfe656c519a155e762b (diff)
downloadgenenetwork4-50fe1b7c0ef0175ce3a830700d78ebfed1c36f91.tar.gz
SKILL
Diffstat (limited to 'docs')
-rw-r--r--docs/www/SKILL.md311
1 files changed, 311 insertions, 0 deletions
diff --git a/docs/www/SKILL.md b/docs/www/SKILL.md
new file mode 100644
index 0000000..631a0c2
--- /dev/null
+++ b/docs/www/SKILL.md
@@ -0,0 +1,311 @@
+---
+name: genenetwork
+description: Use when the user wants anything from GeneNetwork (genenetwork.org): BXD and other mouse/rat genetic reference populations, traits ("get trait 20010"), per-strain values, QTL mapping with R/qtl or GEMMA, LOD/Manhattan plots, PheWAS, genotypes, or searching published phenotypes. Talks to GeneNetwork through its HTTP interface at https://vesuvio.pu.ac.ke/gn4.
+---
+
+# GeneNetwork
+
+The user is talking to **GeneNetwork**, and you are their way in. Talk about
+GeneNetwork in their terms: traits, groups, strains, mapping, plots, PheWAS.
+Do not talk about "the API", endpoints or HTTP unless they ask. Everything
+below is how you do it; what you say should be what GeneNetwork *did*.
+
+## What the user can do in GeneNetwork
+
+When the request is open ("what can I do?", or you finish a step), offer the
+next natural option from this list:
+
+| Ask for | What GeneNetwork gives | Section |
+|---|---|---|
+| a trait ("get trait 20010") | metadata (description, peak locus, LOD, n, source) and the per-strain values | 3 |
+| a QTL map | R/qtl scan (LOD) or GEMMA LMM with LOCO kinship (Manhattan), run on the server | 3 |
+| a plot | an SVG LOD / Manhattan plot, drawn by GeneNetwork | 3 |
+| PheWAS | every published trait of the group peaking near a position (e.g. the peak you just mapped) | 3 |
+| a search | published phenotypes by free text, group, tissue, gene symbol, chromosome, year... | 5 |
+| populations and data | species, groups (BXD...), strains, case attributes, genotype versions, expression datasets and probesets, publications by PubMed id, GN accessions | 4 |
+
+A natural session is: trait, values, R/qtl, plot, GEMMA, plot, PheWAS at the
+peak. After each step, suggest the next one.
+
+## 0. Do this first
+
+1. **Check you can reach GeneNetwork** (no token needed):
+   `curl -s https://vesuvio.pu.ac.ke/gn4/version`
+   and, for the docs repo, `git ls-remote https://git.genenetwork.org/genenetwork4 HEAD`.
+   A sandbox proxy blocks any host that is not on the person's allowlist
+   (`connect_rejected`, or `CONNECT tunnel failed, 403` from git). **Ask them to
+   allow both domains in one go**, not one at a time:
+
+   > To work with GeneNetwork I need two domains allowed for this session:
+   > `vesuvio.pu.ac.ke` (the data and mapping service) and `git.genenetwork.org`
+   > (the docs, so I can reload this skill when it changes). Please add them under
+   > claude.ai > Settings > Capabilities (admins: Organization settings >
+   > Capabilities), then start a new message or session.
+
+   If only the repo is blocked, GeneNetwork itself still works: carry on, and
+   tell the user the skill file cannot be refreshed (section 7). Say all this once
+   and stop probing other routes.
+2. **Token.** Look for one already given to you in the task context. If there
+   is none, stop and ask (section 6). Never guess, invent, reuse a token that
+   was not given to you, or look for another way in: `POST /gn4/get-test-token`
+   is a deliberate 404, and there is no sign-up or login. A token carries no
+   rights of its own; what you may read is decided per dataset.
+3. **Note the build** (section 7): remember `.data.git` from `/version` as
+   your baseline; if it changes later in the session, suggest reloading
+   `docs/www/SKILL.md`.
+4. **Read the spec, not only this page.** `GET /gn4/openapi.json` lists every
+   operation and is what the server really runs; this page can lag. Use
+   `jq -r '.paths | keys[]'` before saying something is not available. The
+descriptions inside it are the real reference: every request option is
+documented in `components.schemas` (for example
+`jq '.components.schemas.RqtlRequest.properties' openapi.json`).
+5. **Assume, don't ask.** A bare numeric trait id (`20010`) is a published
+   phenotype of group **BXD** (dataset `BXDPublish`). Try it; ask only on 404.
+
+Base URL for everything: `https://vesuvio.pu.ac.ke/gn4`. `/gn4` without the
+trailing slash redirects there; anything outside `/gn4/` is a 404 from the
+front end, not from GeneNetwork. Only `/version` and `/openapi.json` need no
+token; everything else answers 401 (no token) or 404 (not routed).
+
+## 1. Read the replies: GeneNetwork tells you what to do next
+
+Every JSON reply is `{"data","meta","notes","next"}` and sometimes `"actions"`.
+
+* `notes`: read them. They say how many strains were matched, what was run,
+  and caveats.
+* `next`: GET links to related things. Each `href` already includes `/gn4`:
+  resolve against the origin (`https://vesuvio.pu.ac.ke$href`).
+* **`actions`: ready-to-send POSTs.** On a trait's `/values` they are the R/qtl
+  and GEMMA calls with the body already filled in; on a **compute reply** they
+  are the matching plot call (href with `?trait=` already set; post the reply
+  back to it). Use them instead of building calls by hand. Select by href, not
+  position: `jq '.actions[] | select(.href|endswith("/compute/rqtl")) | .body'`.
+* **Every compute reply also has `next`**: a PheWAS link at that run's peak
+  marker. Read `next` and `actions` of each reply before deciding what to do.
+* `meta`: `state` (`complete`/`partial`), `took_ms`, `request_id`, plus
+  `columns`/`num_rows` on tabular payloads.
+* `meta.state: partial` means usable but incomplete; the reason is in `notes`.
+  Do not read a `null` enrichment field as "no such value in GeneNetwork"; read
+  it as "ask again later".
+* Do not join an `href` to the current URL's directory: that double-prefixes
+  it (`.../gn4/gn4/...`).
+
+## 2. Workflow: map a trait and plot it
+
+```sh
+API=https://vesuvio.pu.ac.ke/gn4; AUTH="Authorization: Bearer $TOKEN"; D=$(mktemp -d)
+curl -s -H "$AUTH" $API/groups/BXD/traits/20010                       # metadata
+curl -s -H "$AUTH" $API/groups/BXD/traits/20010/values > $D/v.json    # values + actions
+jq '.actions[]|select(.href|endswith("/compute/rqtl"))|.body' $D/v.json > $D/body.json
+
+# R/qtl. Save the FULL reply to a file (never pipe through head), then plot it.
+curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/body.json \
+  $API/compute/rqtl > $D/rqtl.json
+curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/rqtl.json \
+  -D $D/h.txt -o rqtl.svg "$API/plot/rqtl?trait=BXD::20010"; grep -i x-plot-summary $D/h.txt
+
+# GEMMA: same shape
+jq '.actions[]|select(.href|endswith("/compute/gemma"))|.body' $D/v.json > $D/gbody.json
+curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/gbody.json \
+  "$API/compute/gemma?id=t20010" > $D/gemma.json
+curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/gemma.json \
+  -o gemma.svg "$API/plot/gemma?trait=BXD::20010"
+
+# PheWAS at the peak: follow the compute reply's own `next` (10 kb window), or widen it
+jq -r '.next[0].href' $D/rqtl.json
+curl -s -H "$AUTH" --get --data-urlencode locus=<peak marker> --data window=1000000 \
+  $API/groups/BXD/phewas              # window = half-width in bp
+```
+
+Give the user the SVGs (they render directly) and quote the `x-plot-summary`
+header (peak and, for GEMMA, the Bonferroni line).
+
+## 3. GeneNetwork functionality, in detail
+
+**Mapping and plots (POST, verified working)**
+
+* `/compute/rqtl`: R/qtl `scanone` (Haley-Knott) over the group's genotypes.
+  Columns `name, chr, pos, lod`. `nperm=0` by default, so no permutation lines.
+* `/compute/gemma`: GEMMA LMM with leave-one-chromosome-out kinship. Columns
+  `chr, rs, ps, ..., p_lrt`. Pass `?id=` and poll `/compute/gemma/status/{id}`
+  for long runs. Per-marker p-values only; the plot draws a Bonferroni line.
+* `/plot/rqtl`, `/plot/gemma`: post the compute reply back, receive an
+  `image/svg+xml` plot. Do not draw your own.
+* Options (all optional, add to the posted body; full list in
+  `components.schemas.RqtlRequest` / `GemmaRequest`):
+  * R/qtl: `nperm` (permutations; **100 takes about 14 s** and adds
+    `significant`/`suggestive` thresholds that `/plot/rqtl` then draws),
+    `method` (`hk` default, `ehk`, `mr`, `em`, `imp`), `model` (`normal`
+    default, `np`), `control` (a marker to condition on), `genotype_version`,
+    `include_parents_f1`, `covariates`.
+  * GEMMA: `lmm` (default 9), `genotype_version`, `include_parents_f1`,
+    `covariates`. `nperm` is refused (`422 permutations_not_supported`): for
+    significance send the same values to `/compute/rqtl` with `nperm`.
+  * `covariates: [{"case_attribute": "<name>"}]` uses a case attribute of the
+    group; get the valid names from `/groups/{g}/case-attributes` (`.data.names`;
+    BXD has `Epoch, RRID, SeqCvge, Status`, **not** Sex, and an unknown name is
+    `422 covariate_unavailable` listing the valid ones).
+  * `genotype_version`: list them with `/groups/{g}/genotypes` (BXD.8 is current).
+* Timings seen: R/qtl about 7 s (1 MB reply), GEMMA about 3 s (1.9 MB), plots
+  instant. They are synchronous; `?id=` only adds progress logging.
+* Progress: `GET /compute/{rqtl,gemma}/status/{id}` returns `lines` and a
+  `pointer` to pass back as `peek`; the last line is `COMPLETION_MARKER` or
+  `FAILURE_MARKER: <error>`. It exists only for an `id` you passed on the POST
+  (otherwise `404 run_not_found`).
+* `/plot/*` accepts the whole compute reply or just its `data`; query params
+  `trait`, `title`, and `bonferroni=false` (GEMMA only).
+* `/compute/correlation` is a stub with canned data. Do not use it.
+* You never supply genotypes: the server uses its own files.
+
+**PheWAS (GET)** `/groups/{group}/phewas?locus=<marker>` or `?chr=&mb=`,
+`window` (bp, default 10000), `genotype_version`. Returns published traits whose
+recorded peak marker lies in the window, strongest first. An empty list is a
+real answer.
+
+**Data (GET, all need the token)**
+
+| path | gives you |
+|---|---|
+| `/species`, `/species/{s}`, `/species/{s}/datasets` | species, expression datasets |
+| `/groups?species=`, `/groups/{g}`, `/groups/{g}/strains`, `/groups/{g}/datasets`, `/groups/{g}/case-attributes` | populations and samples |
+| `/groups/{g}/traits`, `/groups/{g}/traits/{id}`, `/groups/{g}/traits/{id}/values` | published traits (the first is the bulk list) |
+| `/groups/{g}/traits/{id}/case-attributes/{attribute}` | a trait's case-attribute values |
+| `/groups/{g}/genotypes[/{version}[/markers]]` | genotype versions and markers |
+| `/datasets/{ds}[/probesets[/{p}[/values]]]`, `/traits/{ds}[/{t}[/values]]` | expression data |
+| `/publications/{pubmed_id}/traits` | traits a paper reports (PubMed id, not `publication_id`) |
+| `/accessions/{acc}`, `/mapping-methods` | GN accession lookup, methods offered |
+
+**Bulk jobs.** Fetch `/groups/{g}/traits` once (the whole PublishXRef list:
+`id`, `phenotype_id`, `dataset`, `abbreviation`, `nsamples`; about 15 000 for
+BXD), then `.../traits/{id}/values` per trait, a few at a time. Prefer one bulk
+call to a thousand small ones; do not use `/search` for bulk work.
+
+**Other examples:** `/datasets/Br_U_0803_M/probesets/100001_at` (one expression
+trait), `/publications/2760876/traits` (PubMed id), and
+`/search?q=cerebellum AND group:BXD&limit=5`.
+
+## 4. Gotchas that cost time
+
+* Compute replies are big (about 21 000 rows; GEMMA about 2 MB). Save to a file,
+  inspect with `jq` / `python3 -I`. Piping to `head` forces a rerun.
+* Rows are positional; names are in `meta.columns`.
+* Not every value maps: trait 20010's 41 values map as 39, because C57BL/6J and
+  DBA/2J are not in the BXD.8 genotype file, and `include_parents_f1: true` does
+  not change that. `/groups/{g}/strains` flags such samples with
+  `used_for_mapping: false`, which predicts it. `notes` on the reply says how many
+  matched; tell the user.
+* A peak is often a plateau: neighbouring markers share the same LOD / p-value
+  (trait 20010's GEMMA maximum, 5.79, is shared by several markers from 182.6 to
+  184.3 Mb). So the trait record's `locus`, R/qtl's peak, the plot summary and a
+  reply's `next` can each name a different marker of the same strength. That is
+  not an error: say "peak region" and use the marker the reply's `next` gives.
+* PheWAS window matters: at the trait-20010 peak, 10 kb finds 4 traits and 1 Mb
+  finds 19. The reply's `next` offers a ten-times wider window.
+* Public-data rule: `public > 0` and `confidentiality < 1`, or older than five
+  years; otherwise 403, do not retry. Metadata is visible even when values are not.
+* There are no write endpoints, and the search index holds public documents only.
+* Older docs said `POST /compute/*` is refused with 403. On this build it works.
+  If you do get a 403 there, that is the answer: do not work around it. If the
+  task needs private data or computation you are not granted, stop and ask a
+  human: that needs an account and a grant, not a different URL.
+
+## 5. Search
+
+`/search?q=` is Xapian: juxtaposed terms are **OR**, so write `AND`; free text
+is stemmed and case-insensitive; quoted phrases, wildcards and hyphenated words
+match nothing. Fields: exact `group`, `dataset`, `species`, `author`, `tissue`,
+`symbol`, `chr`, `peakchr`, `position`; stemmed `description`, `rif`, `wiki`;
+ranges `mean`, `peak`, `peakmb`, `additive`, `year` (`field:lo..hi`).
+Aliases: `pos` for `position`, `RIF` for `rif`, `WIKI` for `wiki`; `Hs`/`Mm`
+are species-location shorthands. Values are lowercased, so case does not matter.
+A prefix whose value is not in the index matches nothing; that is "no such
+value", not an error. `NEAR`/`ADJ` do not work either. A dangling operator or
+unbalanced quote is `422 invalid_query` with Xapian's reason in `detail`: fix
+it, do not retry. `type=phenotype|gene`, `limit` up to 100, `page`. There is no
+total: counting means paging, and each page costs time proportional to the
+matches left, so do not page through thousands of hits. For bulk work use
+`/groups/{g}/traits`.
+
+`.data` is an array of hits. Each has `name` (the **trait id**, e.g. `"21614"`),
+`dataset`, `dataset_fullname`, `group`, `species`, `authors`, `year`,
+`description`, `mean` and more. Without `group:` you also get other groups'
+phenotypes (for example `C57BL-6JxC57BL-6NJF2Publish`), whose ids only mean
+something inside their own group; add `AND group:BXD` to stay in BXD.
+
+## 6. Tokens and errors
+
+A GeneNetwork team member issues the token by hand; it cannot be self-served.
+If you have none, ask: "To read GeneNetwork data I need a GeneNetwork API token
+(a JWT) for https://vesuvio.pu.ac.ke/gn4. It is issued by hand; it helps to say
+who it is for, which groups/datasets, how long (default 30 days) and what for."
+Ask the user to pass it out of band, not in a public issue, a commit or a
+shared chat. Send it as `Authorization: Bearer <token>` on every request except
+`/version` and `/openapi.json`, and keep it for the whole session. Never put it
+in a URL, a commit, a log line or anything shown to someone else, and do not
+hand it to other users or paste it into shared state.
+
+| status | do |
+|---|---|
+| 401 `invalid_token` | `detail` says `ExpiredSignature` (expired) or `InvalidIssuer` (wrong deployment). You cannot renew it: tell the user, ask for a fresh one quoting the `request_id`, retry **once**, then stop and report; do not loop or look for another credential |
+| 403 `forbidden` | not public: do not retry |
+| 404 `not_found` | check the name; `next` lists valid values |
+| 422 `invalid_query`, `invalid_lmm` | fix the query or parameter; do not retry |
+| 429 | back off and honour `Retry-After` (not emitted today, but in the contract) |
+| 502/503 | transient: back off, max about 5 tries |
+
+Errors are RFC 7807-style JSON (`type`, `title`, `status`, `code`, `detail`,
+`operation_id`, `spec_url`, `request_id`, sometimes `next`); `type`, `spec_url`
+and `next` are absolute paths under `/gn4/`. Quote `request_id` when reporting
+a problem (also in the `x-request-id` header). No rate limit is configured, but
+be considerate: a few concurrent requests, no tight polling loops.
+
+## 7. Keep this skill current: reload it if GeneNetwork changes mid-session
+
+Do not rely on a version written into this file. Instead, **remember the build
+you saw at the start of the session** and compare against it later.
+
+1. At the start (section 0), note `curl -s https://vesuvio.pu.ac.ke/gn4/version
+   | jq -r .data.git` as your session baseline.
+2. Check `/version` again when something behaves differently from what is
+   written here (an unexpected 404, a new field in `actions`, a path in
+   `openapi.json` this file does not mention), and before a long or repeated
+   piece of work.
+3. If the revision is no longer the baseline, GeneNetwork was updated while you
+   worked. **Suggest to the user that you reload SKILL.md from the repo**
+   (`docs/www/SKILL.md` in `https://git.genenetwork.org/genenetwork4`; a sparse
+   clone of `docs` is enough), and use that copy from then on. One line is
+   enough, for example: "GeneNetwork was updated since we started (<old> ->
+   <new>); I'd reload SKILL.md from docs/www/ to pick up any changes. Shall I?"
+   Then set the new revision as your baseline.
+4. If the repo is unreachable from the sandbox, fall back to
+   `/gn4/openapi.json`, which always matches the running server, and tell the
+   user the skill file itself could not be refreshed.
+
+## 8. Reference docs
+
+The docs live in the public repo and are the authoritative version:
+
+```sh
+git clone --depth 1 --sparse https://git.genenetwork.org/genenetwork4
+cd genenetwork4 && git sparse-checkout set docs
+```
+
+About 2.5 MB; drop `--sparse` for the whole tree. Then read:
+
+| file | what it is |
+|---|---|
+| `docs/www/SKILL.md` | this page |
+| `docs/search-engines.md` | how `/search` works, engines, quirks |
+| `README.md` | every endpoint, parameter and response shape |
+| `docs/nginx.md`, `docs/www/operator-checklist.md` | operator-level; you do not need them |
+
+**Clone; do not fetch the web UI.** `git.genenetwork.org/.../tree/...` pages sit
+behind a proof-of-work challenge (Anubis) and the raw endpoint answers 403, so
+a plain fetch returns an HTML challenge page, not Markdown. If a fetch comes
+back as HTML mentioning Anubis, stop, do not retry, and use the copy you have.
+The git protocol itself passes. Sandboxes sometimes cannot reach it either;
+`/gn4/openapi.json` is the fallback.
+
+To compare revisions, `git -C genenetwork4 log -1 --format=%h` against `.data.git`
+from `/version`: they need not match exactly (docs can be newer than the
+running build), but a large gap means the server may lack what the docs describe.