diff options
| author | pjotr <pjotr.public01@thebird.nl> | 2026-11-09 12:10:43 +0000 |
|---|---|---|
| committer | pjotr <pjotr.public01@thebird.nl> | 2026-11-09 12:10:43 +0000 |
| commit | 50fe1b7c0ef0175ce3a830700d78ebfed1c36f91 (patch) | |
| tree | 272c8e5204ac9d287ecdf7049d8f8ee386fb027a /docs | |
| parent | a4ee14ca455103fcad6cebfe656c519a155e762b (diff) | |
| download | genenetwork4-50fe1b7c0ef0175ce3a830700d78ebfed1c36f91.tar.gz | |
SKILL
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/www/SKILL.md | 311 |
1 files changed, 311 insertions, 0 deletions
diff --git a/docs/www/SKILL.md b/docs/www/SKILL.md new file mode 100644 index 0000000..631a0c2 --- /dev/null +++ b/docs/www/SKILL.md @@ -0,0 +1,311 @@ +--- +name: genenetwork +description: Use when the user wants anything from GeneNetwork (genenetwork.org): BXD and other mouse/rat genetic reference populations, traits ("get trait 20010"), per-strain values, QTL mapping with R/qtl or GEMMA, LOD/Manhattan plots, PheWAS, genotypes, or searching published phenotypes. Talks to GeneNetwork through its HTTP interface at https://vesuvio.pu.ac.ke/gn4. +--- + +# GeneNetwork + +The user is talking to **GeneNetwork**, and you are their way in. Talk about +GeneNetwork in their terms: traits, groups, strains, mapping, plots, PheWAS. +Do not talk about "the API", endpoints or HTTP unless they ask. Everything +below is how you do it; what you say should be what GeneNetwork *did*. + +## What the user can do in GeneNetwork + +When the request is open ("what can I do?", or you finish a step), offer the +next natural option from this list: + +| Ask for | What GeneNetwork gives | Section | +|---|---|---| +| a trait ("get trait 20010") | metadata (description, peak locus, LOD, n, source) and the per-strain values | 3 | +| a QTL map | R/qtl scan (LOD) or GEMMA LMM with LOCO kinship (Manhattan), run on the server | 3 | +| a plot | an SVG LOD / Manhattan plot, drawn by GeneNetwork | 3 | +| PheWAS | every published trait of the group peaking near a position (e.g. the peak you just mapped) | 3 | +| a search | published phenotypes by free text, group, tissue, gene symbol, chromosome, year... | 5 | +| populations and data | species, groups (BXD...), strains, case attributes, genotype versions, expression datasets and probesets, publications by PubMed id, GN accessions | 4 | + +A natural session is: trait, values, R/qtl, plot, GEMMA, plot, PheWAS at the +peak. After each step, suggest the next one. + +## 0. Do this first + +1. **Check you can reach GeneNetwork** (no token needed): + `curl -s https://vesuvio.pu.ac.ke/gn4/version` + and, for the docs repo, `git ls-remote https://git.genenetwork.org/genenetwork4 HEAD`. + A sandbox proxy blocks any host that is not on the person's allowlist + (`connect_rejected`, or `CONNECT tunnel failed, 403` from git). **Ask them to + allow both domains in one go**, not one at a time: + + > To work with GeneNetwork I need two domains allowed for this session: + > `vesuvio.pu.ac.ke` (the data and mapping service) and `git.genenetwork.org` + > (the docs, so I can reload this skill when it changes). Please add them under + > claude.ai > Settings > Capabilities (admins: Organization settings > + > Capabilities), then start a new message or session. + + If only the repo is blocked, GeneNetwork itself still works: carry on, and + tell the user the skill file cannot be refreshed (section 7). Say all this once + and stop probing other routes. +2. **Token.** Look for one already given to you in the task context. If there + is none, stop and ask (section 6). Never guess, invent, reuse a token that + was not given to you, or look for another way in: `POST /gn4/get-test-token` + is a deliberate 404, and there is no sign-up or login. A token carries no + rights of its own; what you may read is decided per dataset. +3. **Note the build** (section 7): remember `.data.git` from `/version` as + your baseline; if it changes later in the session, suggest reloading + `docs/www/SKILL.md`. +4. **Read the spec, not only this page.** `GET /gn4/openapi.json` lists every + operation and is what the server really runs; this page can lag. Use + `jq -r '.paths | keys[]'` before saying something is not available. The +descriptions inside it are the real reference: every request option is +documented in `components.schemas` (for example +`jq '.components.schemas.RqtlRequest.properties' openapi.json`). +5. **Assume, don't ask.** A bare numeric trait id (`20010`) is a published + phenotype of group **BXD** (dataset `BXDPublish`). Try it; ask only on 404. + +Base URL for everything: `https://vesuvio.pu.ac.ke/gn4`. `/gn4` without the +trailing slash redirects there; anything outside `/gn4/` is a 404 from the +front end, not from GeneNetwork. Only `/version` and `/openapi.json` need no +token; everything else answers 401 (no token) or 404 (not routed). + +## 1. Read the replies: GeneNetwork tells you what to do next + +Every JSON reply is `{"data","meta","notes","next"}` and sometimes `"actions"`. + +* `notes`: read them. They say how many strains were matched, what was run, + and caveats. +* `next`: GET links to related things. Each `href` already includes `/gn4`: + resolve against the origin (`https://vesuvio.pu.ac.ke$href`). +* **`actions`: ready-to-send POSTs.** On a trait's `/values` they are the R/qtl + and GEMMA calls with the body already filled in; on a **compute reply** they + are the matching plot call (href with `?trait=` already set; post the reply + back to it). Use them instead of building calls by hand. Select by href, not + position: `jq '.actions[] | select(.href|endswith("/compute/rqtl")) | .body'`. +* **Every compute reply also has `next`**: a PheWAS link at that run's peak + marker. Read `next` and `actions` of each reply before deciding what to do. +* `meta`: `state` (`complete`/`partial`), `took_ms`, `request_id`, plus + `columns`/`num_rows` on tabular payloads. +* `meta.state: partial` means usable but incomplete; the reason is in `notes`. + Do not read a `null` enrichment field as "no such value in GeneNetwork"; read + it as "ask again later". +* Do not join an `href` to the current URL's directory: that double-prefixes + it (`.../gn4/gn4/...`). + +## 2. Workflow: map a trait and plot it + +```sh +API=https://vesuvio.pu.ac.ke/gn4; AUTH="Authorization: Bearer $TOKEN"; D=$(mktemp -d) +curl -s -H "$AUTH" $API/groups/BXD/traits/20010 # metadata +curl -s -H "$AUTH" $API/groups/BXD/traits/20010/values > $D/v.json # values + actions +jq '.actions[]|select(.href|endswith("/compute/rqtl"))|.body' $D/v.json > $D/body.json + +# R/qtl. Save the FULL reply to a file (never pipe through head), then plot it. +curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/body.json \ + $API/compute/rqtl > $D/rqtl.json +curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/rqtl.json \ + -D $D/h.txt -o rqtl.svg "$API/plot/rqtl?trait=BXD::20010"; grep -i x-plot-summary $D/h.txt + +# GEMMA: same shape +jq '.actions[]|select(.href|endswith("/compute/gemma"))|.body' $D/v.json > $D/gbody.json +curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/gbody.json \ + "$API/compute/gemma?id=t20010" > $D/gemma.json +curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/gemma.json \ + -o gemma.svg "$API/plot/gemma?trait=BXD::20010" + +# PheWAS at the peak: follow the compute reply's own `next` (10 kb window), or widen it +jq -r '.next[0].href' $D/rqtl.json +curl -s -H "$AUTH" --get --data-urlencode locus=<peak marker> --data window=1000000 \ + $API/groups/BXD/phewas # window = half-width in bp +``` + +Give the user the SVGs (they render directly) and quote the `x-plot-summary` +header (peak and, for GEMMA, the Bonferroni line). + +## 3. GeneNetwork functionality, in detail + +**Mapping and plots (POST, verified working)** + +* `/compute/rqtl`: R/qtl `scanone` (Haley-Knott) over the group's genotypes. + Columns `name, chr, pos, lod`. `nperm=0` by default, so no permutation lines. +* `/compute/gemma`: GEMMA LMM with leave-one-chromosome-out kinship. Columns + `chr, rs, ps, ..., p_lrt`. Pass `?id=` and poll `/compute/gemma/status/{id}` + for long runs. Per-marker p-values only; the plot draws a Bonferroni line. +* `/plot/rqtl`, `/plot/gemma`: post the compute reply back, receive an + `image/svg+xml` plot. Do not draw your own. +* Options (all optional, add to the posted body; full list in + `components.schemas.RqtlRequest` / `GemmaRequest`): + * R/qtl: `nperm` (permutations; **100 takes about 14 s** and adds + `significant`/`suggestive` thresholds that `/plot/rqtl` then draws), + `method` (`hk` default, `ehk`, `mr`, `em`, `imp`), `model` (`normal` + default, `np`), `control` (a marker to condition on), `genotype_version`, + `include_parents_f1`, `covariates`. + * GEMMA: `lmm` (default 9), `genotype_version`, `include_parents_f1`, + `covariates`. `nperm` is refused (`422 permutations_not_supported`): for + significance send the same values to `/compute/rqtl` with `nperm`. + * `covariates: [{"case_attribute": "<name>"}]` uses a case attribute of the + group; get the valid names from `/groups/{g}/case-attributes` (`.data.names`; + BXD has `Epoch, RRID, SeqCvge, Status`, **not** Sex, and an unknown name is + `422 covariate_unavailable` listing the valid ones). + * `genotype_version`: list them with `/groups/{g}/genotypes` (BXD.8 is current). +* Timings seen: R/qtl about 7 s (1 MB reply), GEMMA about 3 s (1.9 MB), plots + instant. They are synchronous; `?id=` only adds progress logging. +* Progress: `GET /compute/{rqtl,gemma}/status/{id}` returns `lines` and a + `pointer` to pass back as `peek`; the last line is `COMPLETION_MARKER` or + `FAILURE_MARKER: <error>`. It exists only for an `id` you passed on the POST + (otherwise `404 run_not_found`). +* `/plot/*` accepts the whole compute reply or just its `data`; query params + `trait`, `title`, and `bonferroni=false` (GEMMA only). +* `/compute/correlation` is a stub with canned data. Do not use it. +* You never supply genotypes: the server uses its own files. + +**PheWAS (GET)** `/groups/{group}/phewas?locus=<marker>` or `?chr=&mb=`, +`window` (bp, default 10000), `genotype_version`. Returns published traits whose +recorded peak marker lies in the window, strongest first. An empty list is a +real answer. + +**Data (GET, all need the token)** + +| path | gives you | +|---|---| +| `/species`, `/species/{s}`, `/species/{s}/datasets` | species, expression datasets | +| `/groups?species=`, `/groups/{g}`, `/groups/{g}/strains`, `/groups/{g}/datasets`, `/groups/{g}/case-attributes` | populations and samples | +| `/groups/{g}/traits`, `/groups/{g}/traits/{id}`, `/groups/{g}/traits/{id}/values` | published traits (the first is the bulk list) | +| `/groups/{g}/traits/{id}/case-attributes/{attribute}` | a trait's case-attribute values | +| `/groups/{g}/genotypes[/{version}[/markers]]` | genotype versions and markers | +| `/datasets/{ds}[/probesets[/{p}[/values]]]`, `/traits/{ds}[/{t}[/values]]` | expression data | +| `/publications/{pubmed_id}/traits` | traits a paper reports (PubMed id, not `publication_id`) | +| `/accessions/{acc}`, `/mapping-methods` | GN accession lookup, methods offered | + +**Bulk jobs.** Fetch `/groups/{g}/traits` once (the whole PublishXRef list: +`id`, `phenotype_id`, `dataset`, `abbreviation`, `nsamples`; about 15 000 for +BXD), then `.../traits/{id}/values` per trait, a few at a time. Prefer one bulk +call to a thousand small ones; do not use `/search` for bulk work. + +**Other examples:** `/datasets/Br_U_0803_M/probesets/100001_at` (one expression +trait), `/publications/2760876/traits` (PubMed id), and +`/search?q=cerebellum AND group:BXD&limit=5`. + +## 4. Gotchas that cost time + +* Compute replies are big (about 21 000 rows; GEMMA about 2 MB). Save to a file, + inspect with `jq` / `python3 -I`. Piping to `head` forces a rerun. +* Rows are positional; names are in `meta.columns`. +* Not every value maps: trait 20010's 41 values map as 39, because C57BL/6J and + DBA/2J are not in the BXD.8 genotype file, and `include_parents_f1: true` does + not change that. `/groups/{g}/strains` flags such samples with + `used_for_mapping: false`, which predicts it. `notes` on the reply says how many + matched; tell the user. +* A peak is often a plateau: neighbouring markers share the same LOD / p-value + (trait 20010's GEMMA maximum, 5.79, is shared by several markers from 182.6 to + 184.3 Mb). So the trait record's `locus`, R/qtl's peak, the plot summary and a + reply's `next` can each name a different marker of the same strength. That is + not an error: say "peak region" and use the marker the reply's `next` gives. +* PheWAS window matters: at the trait-20010 peak, 10 kb finds 4 traits and 1 Mb + finds 19. The reply's `next` offers a ten-times wider window. +* Public-data rule: `public > 0` and `confidentiality < 1`, or older than five + years; otherwise 403, do not retry. Metadata is visible even when values are not. +* There are no write endpoints, and the search index holds public documents only. +* Older docs said `POST /compute/*` is refused with 403. On this build it works. + If you do get a 403 there, that is the answer: do not work around it. If the + task needs private data or computation you are not granted, stop and ask a + human: that needs an account and a grant, not a different URL. + +## 5. Search + +`/search?q=` is Xapian: juxtaposed terms are **OR**, so write `AND`; free text +is stemmed and case-insensitive; quoted phrases, wildcards and hyphenated words +match nothing. Fields: exact `group`, `dataset`, `species`, `author`, `tissue`, +`symbol`, `chr`, `peakchr`, `position`; stemmed `description`, `rif`, `wiki`; +ranges `mean`, `peak`, `peakmb`, `additive`, `year` (`field:lo..hi`). +Aliases: `pos` for `position`, `RIF` for `rif`, `WIKI` for `wiki`; `Hs`/`Mm` +are species-location shorthands. Values are lowercased, so case does not matter. +A prefix whose value is not in the index matches nothing; that is "no such +value", not an error. `NEAR`/`ADJ` do not work either. A dangling operator or +unbalanced quote is `422 invalid_query` with Xapian's reason in `detail`: fix +it, do not retry. `type=phenotype|gene`, `limit` up to 100, `page`. There is no +total: counting means paging, and each page costs time proportional to the +matches left, so do not page through thousands of hits. For bulk work use +`/groups/{g}/traits`. + +`.data` is an array of hits. Each has `name` (the **trait id**, e.g. `"21614"`), +`dataset`, `dataset_fullname`, `group`, `species`, `authors`, `year`, +`description`, `mean` and more. Without `group:` you also get other groups' +phenotypes (for example `C57BL-6JxC57BL-6NJF2Publish`), whose ids only mean +something inside their own group; add `AND group:BXD` to stay in BXD. + +## 6. Tokens and errors + +A GeneNetwork team member issues the token by hand; it cannot be self-served. +If you have none, ask: "To read GeneNetwork data I need a GeneNetwork API token +(a JWT) for https://vesuvio.pu.ac.ke/gn4. It is issued by hand; it helps to say +who it is for, which groups/datasets, how long (default 30 days) and what for." +Ask the user to pass it out of band, not in a public issue, a commit or a +shared chat. Send it as `Authorization: Bearer <token>` on every request except +`/version` and `/openapi.json`, and keep it for the whole session. Never put it +in a URL, a commit, a log line or anything shown to someone else, and do not +hand it to other users or paste it into shared state. + +| status | do | +|---|---| +| 401 `invalid_token` | `detail` says `ExpiredSignature` (expired) or `InvalidIssuer` (wrong deployment). You cannot renew it: tell the user, ask for a fresh one quoting the `request_id`, retry **once**, then stop and report; do not loop or look for another credential | +| 403 `forbidden` | not public: do not retry | +| 404 `not_found` | check the name; `next` lists valid values | +| 422 `invalid_query`, `invalid_lmm` | fix the query or parameter; do not retry | +| 429 | back off and honour `Retry-After` (not emitted today, but in the contract) | +| 502/503 | transient: back off, max about 5 tries | + +Errors are RFC 7807-style JSON (`type`, `title`, `status`, `code`, `detail`, +`operation_id`, `spec_url`, `request_id`, sometimes `next`); `type`, `spec_url` +and `next` are absolute paths under `/gn4/`. Quote `request_id` when reporting +a problem (also in the `x-request-id` header). No rate limit is configured, but +be considerate: a few concurrent requests, no tight polling loops. + +## 7. Keep this skill current: reload it if GeneNetwork changes mid-session + +Do not rely on a version written into this file. Instead, **remember the build +you saw at the start of the session** and compare against it later. + +1. At the start (section 0), note `curl -s https://vesuvio.pu.ac.ke/gn4/version + | jq -r .data.git` as your session baseline. +2. Check `/version` again when something behaves differently from what is + written here (an unexpected 404, a new field in `actions`, a path in + `openapi.json` this file does not mention), and before a long or repeated + piece of work. +3. If the revision is no longer the baseline, GeneNetwork was updated while you + worked. **Suggest to the user that you reload SKILL.md from the repo** + (`docs/www/SKILL.md` in `https://git.genenetwork.org/genenetwork4`; a sparse + clone of `docs` is enough), and use that copy from then on. One line is + enough, for example: "GeneNetwork was updated since we started (<old> -> + <new>); I'd reload SKILL.md from docs/www/ to pick up any changes. Shall I?" + Then set the new revision as your baseline. +4. If the repo is unreachable from the sandbox, fall back to + `/gn4/openapi.json`, which always matches the running server, and tell the + user the skill file itself could not be refreshed. + +## 8. Reference docs + +The docs live in the public repo and are the authoritative version: + +```sh +git clone --depth 1 --sparse https://git.genenetwork.org/genenetwork4 +cd genenetwork4 && git sparse-checkout set docs +``` + +About 2.5 MB; drop `--sparse` for the whole tree. Then read: + +| file | what it is | +|---|---| +| `docs/www/SKILL.md` | this page | +| `docs/search-engines.md` | how `/search` works, engines, quirks | +| `README.md` | every endpoint, parameter and response shape | +| `docs/nginx.md`, `docs/www/operator-checklist.md` | operator-level; you do not need them | + +**Clone; do not fetch the web UI.** `git.genenetwork.org/.../tree/...` pages sit +behind a proof-of-work challenge (Anubis) and the raw endpoint answers 403, so +a plain fetch returns an HTML challenge page, not Markdown. If a fetch comes +back as HTML mentioning Anubis, stop, do not retry, and use the copy you have. +The git protocol itself passes. Sandboxes sometimes cannot reach it either; +`/gn4/openapi.json` is the fallback. + +To compare revisions, `git -C genenetwork4 log -1 --format=%h` against `.data.git` +from `/version`: they need not match exactly (docs can be newer than the +running build), but a large gap means the server may lack what the docs describe. |
