--- name: genenetwork description: Use when the user wants anything from GeneNetwork (genenetwork.org): BXD and other mouse/rat genetic reference populations, traits ("get trait 20010"), per-strain values, QTL mapping with R/qtl or GEMMA, LOD/Manhattan plots, PheWAS, genotypes, or searching published phenotypes. Talks to GeneNetwork through its HTTP interface at https://vesuvio.pu.ac.ke/gn4. --- # GeneNetwork The user is talking to **GeneNetwork**, and you are their way in. Talk about GeneNetwork in their terms: traits, groups, strains, mapping, plots, PheWAS. Do not talk about "the API", endpoints or HTTP unless they ask. Everything below is how you do it; what you say should be what GeneNetwork *did*. ## What the user can do in GeneNetwork When the request is open ("what can I do?", or you finish a step), offer the next natural option from this list: | Ask for | What GeneNetwork gives | Section | |---|---|---| | a trait ("get trait 20010") | metadata (description, peak locus, LOD, n, source) and the per-strain values | 3 | | a QTL map | R/qtl scan (LOD) or GEMMA LMM with LOCO kinship (Manhattan), run on the server | 3 | | a plot | an SVG LOD / Manhattan plot, drawn by GeneNetwork | 3 | | PheWAS | every published trait of the group peaking near a position (e.g. the peak you just mapped) | 3 | | a search | published phenotypes by free text, group, tissue, gene symbol, chromosome, year... | 5 | | populations and data | species, groups (BXD...), strains, case attributes, genotype versions, expression datasets and probesets, publications by PubMed id, GN accessions | 4 | A natural session is: trait, values, R/qtl, plot, GEMMA, plot, PheWAS at the peak — and `/compute/fitqtl` at that peak when the user asks how much it matters (with a covariate, if one could confound it). Whenever the group records metadata that could confound the trait (sex, age, epoch, batch, sequencing depth), the same mappings again with that covariate fitted, to see the peak survive it. After each step, suggest the next one. ## 0. Do this first 1. **Check you can reach GeneNetwork** (no token needed): `curl -s https://vesuvio.pu.ac.ke/gn4/version` and, for the docs repo, `git ls-remote https://git.genenetwork.org/genenetwork4 HEAD`. A sandbox proxy blocks any host that is not on the person's allowlist (`connect_rejected`, or `CONNECT tunnel failed, 403` from git). **Ask them to allow both domains in one go**, not one at a time: > To work with GeneNetwork I need two domains allowed for this session: > `vesuvio.pu.ac.ke` (the data and mapping service) and `git.genenetwork.org` > (the docs, so I can reload this skill when it changes). Please add them under > claude.ai > Settings > Capabilities (admins: Organization settings > > Capabilities), then start a new message or session. If only the repo is blocked, GeneNetwork itself still works: carry on, and tell the user the skill file cannot be refreshed (section 7). Say all this once and stop probing other routes. 2. **Token.** Look for one already given to you in the task context. If there is none, stop and ask (section 6). Never guess, invent, reuse a token that was not given to you, or look for another way in: `POST /gn4/get-test-token` is a deliberate 404, and there is no sign-up or login. A token carries no rights of its own; what you may read is decided per dataset. 3. **Note the build** (section 7): remember `.data.git` from `/version` as your baseline; if it changes later in the session, suggest reloading `docs/www/SKILL.md`. 4. **Read the spec, not only this page.** `GET /gn4/openapi.json` lists every operation and is what the server really runs; this page can lag. Use `jq -r '.paths | keys[]'` before saying something is not available. The descriptions inside it are the real reference: every request option is documented in `components.schemas` (for example `jq '.components.schemas.RqtlRequest.properties' openapi.json`). 5. **Assume, don't ask.** A bare numeric trait id (`20010`) is a published phenotype of group **BXD** (dataset `BXDPublish`). Try it; ask only on 404. Base URL for everything: `https://vesuvio.pu.ac.ke/gn4`. `/gn4` without the trailing slash redirects there; anything outside `/gn4/` is a 404 from the front end, not from GeneNetwork. Only `/version` and `/openapi.json` need no token; everything else answers 401 (no token) or 404 (not routed). ## 1. Read the replies: GeneNetwork tells you what to do next Every JSON reply is `{"data","meta","notes","next"}` and sometimes `"actions"`. * `notes`: read them. They say how many strains were matched, what was run, and caveats. * `next`: GET links to related things. Each `href` already includes `/gn4`: resolve against the origin (`https://vesuvio.pu.ac.ke$href`). * **`actions`: ready-to-send POSTs.** On a trait's `/values` they are the R/qtl and GEMMA calls with the body already filled in; on a **compute reply** they are the matching plot call (href with `?trait=` already set; post the reply back to it). Use them instead of building calls by hand. Select by href, not position: `jq '.actions[] | select(.href|endswith("/compute/rqtl")) | .body'`. * **Every compute reply also has `next`**: a PheWAS link at that run's peak marker. Read `next` and `actions` of each reply before deciding what to do. * `meta`: `state` (`complete`/`partial`), `took_ms`, `request_id`, plus `columns`/`num_rows` on tabular payloads. * `meta.state: partial` means usable but incomplete; the reason is in `notes`. Do not read a `null` enrichment field as "no such value in GeneNetwork"; read it as "ask again later". * Do not join an `href` to the current URL's directory: that double-prefixes it (`.../gn4/gn4/...`). ### Always offer the next step **End every answer with one sentence saying what to do next**, built from that reply's own `next`/`actions` — never from an idea of what a mapping pipeline "usually" does. The server already computed the options; the sentence is you relaying them. * **One sentence, after every job** — a compute run, a lookup, a plot, an upload, a basket edit. Not a list, not a section: *"Next: plot that scan (`POST /plot/rqtl` with the reply) or look at the PheWAS at rsm10000000888 (`GET /groups/BXD/phewas?locus=rsm10000000888`)."* * **Prefer the reply's `actions` href** over a hand-built call, and say which one you are offering, so the user can paste it or ask for it. * **A refusal is an answer too.** When the reply is an error, its `next` or its detail names the alternative (`/compute/bulklmm` for a GEMMA permutation, `/compute/gemma` for a covariate fit): offer that as the sentence. * **Do not stack suggestions.** One `next`, plus one alternative at most, and only if they lead somewhere genuinely different. * **The user can switch it off.** "Skip the suggestions", "just the result", "no next steps" — then stop offering them for the rest of the session (or until they say otherwise), and give the answer alone. Do not re-add a suggestion in later sentences; it is the sentence that is disabled, not the content. * **Never invent a step the API cannot do.** If nothing in `next`/`actions` fits, say so ("nothing further from this reply") rather than proposing a path that does not exist. ## 2. Workflow: map a trait and plot it ```sh API=https://vesuvio.pu.ac.ke/gn4; AUTH="Authorization: Bearer $TOKEN"; D=$(mktemp -d) curl -s -H "$AUTH" $API/groups/BXD/traits/20010 # metadata curl -s -H "$AUTH" $API/groups/BXD/traits/20010/values > $D/v.json # values + actions jq '.actions[]|select(.href|endswith("/compute/rqtl"))|.body' $D/v.json > $D/body.json # R/qtl. Save the FULL reply to a file (never pipe through head), then plot it. curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/body.json \ $API/compute/rqtl > $D/rqtl.json curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/rqtl.json \ -D $D/h.txt -o rqtl.svg "$API/plot/rqtl?trait=BXD::20010"; grep -i x-plot-summary $D/h.txt # GEMMA: same shape. Use /compute/gemma/ravanan -- it is the default runner # (see "Which GEMMA endpoint" below). jq '.actions[]|select(.href|endswith("/compute/gemma"))|.body' $D/v.json > $D/gbody.json curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/gbody.json \ "$API/compute/gemma/ravanan?id=t20010" > $D/gemma.json curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/gemma.json \ -o gemma.svg "$API/plot/gemma?trait=BXD::20010" # PheWAS at the peak: follow the compute reply's own `next` (10 kb window), or widen it jq -r '.next[0].href' $D/rqtl.json curl -s -H "$AUTH" --get --data-urlencode locus= --data window=1000000 \ $API/groups/BXD/phewas # window = half-width in bp ``` Give the user the SVGs (they render directly) and quote the `x-plot-summary` header (peak, and the reference lines drawn when you asked for them). ### Short API tokens (`gn4ai_...`) A second credential format beside gn-auth's JWT, for agents and scripts that should not carry a user's password or a long-lived JWT. Opaque: nothing is stored inside it, and everything about it lives in the server's registry (one SQLite database, `tokens.sqlite`). - **Administer them from this machine only.** `scripts/gn4-token.py create|list|revoke|who` (routes `/tokens`, `/tokens/{id}`, `/tokens/who`). Those routes answer only a direct caller on this machine — the server checks the TCP peer address, not a header — and a request that arrives through a proxy is refused, so this can never be driven from the network. - **The token is shown exactly once**, by `create`. Only its SHA-256 is stored; there is no way to recover or enumerate one. `list` shows id, subject, scope, status, prefix, expiry and last use. - **Send it as `Authorization: Bearer gn4ai_...`** — never as a query parameter, which would land in nginx's log and in `next` links. - **Public data only.** It works on every token-only endpoint (search, compute, plots, genotype metadata and frequencies, baskets, uploads, matrices) and on secured endpoints for public rows; a confidential dataset answers 403 telling the caller to use a gn-auth JWT, because gn-auth cannot validate an opaque token. - **Revocation takes effect on the next request**, with no restart: it is a row in the registry. Expiry is fixed when minted (30 days by default). - 64 characters, `gn4ai_` + 56 body characters + a 2-character checksum, from an alphabet with no look-alikes (`0/O`, `1/l/I`) — a typo is refused before the registry is asked. - **In the log**, one JSON line per request names the person and the credential: `{"event":"request","ts":…,"user":"zachs","scheme":"opaque","token":"agent-1"}` — never the token or its hash. `scripts/gn4-log.py -f LOG` renders that as a table with the handle resolved against the registry, so it shows the subject as the registry has it now and the token's status *today* (`[revoked]`, `[expired]`, `[unknown]`); `--json` enriches the records for `jq`, and `--summary` counts requests per caller and credential. ### Which GEMMA endpoint: ravanan by default Two endpoints run the same computation, and the reply shape is identical: | endpoint | runs | |---|---| | `/compute/gemma/ravanan` | `gemma` itself, one invocation per chromosome, scattered by ravanan (Slurm) | | `/compute/gemma` | `gemma-wrapper` (kinship, then GWA, both `--loco`) as a server subprocess, no cluster | **Use `/compute/gemma/ravanan` by default.** Reach for `/compute/gemma` only when: - **Slurm is down or job submission fails** (`502`/`ravanan_failed`, or a job that never leaves `PENDING`) -- the in-process runner needs no cluster; - **the user asks for it**, or the request needs something only it supports (covariates, normalization -- ravanan refuses those with a `422`); - **you are testing or verifying results.** The wrapper endpoint is the reference: send the same body to both and compare rows. `scripts/gemma-compare.py --base URL --body FILE` does exactly that, prints both wall-clock times, and exits non-zero if any `rs` disagrees on `beta`, `se`, `l_mle` or `p_lrt`. Both are warm-cached and both are cheap to repeat: a repeat of an identical request comes back from cache (ravanan's store, or gemma-wrapper's `--cache-dir`) in ~1-2 s against ~12-15 s for a first run, so re-running to verify costs almost nothing. ## 3. GeneNetwork functionality, in detail **Mapping and plots (POST, verified working)** * `/compute/rqtl`: R/qtl `scanone` (Haley-Knott) over the group's genotypes. Columns `name, chr, pos, lod`. `nperm=0` by default, so no permutation lines. * `/compute/gemma`: GEMMA LMM with leave-one-chromosome-out kinship. Columns `chr, rs, ps, n_miss, allele1, allele0, af, beta, se, l_mle, additive, p_lrt`. Pass `?id=` and poll `/compute/gemma/status/{id}` for long runs. Per-marker p-values only; the plot draws no reference line unless asked — `?bonferroni=true` for GEMMA's own, `?thresholds=` for another engine's. **Effect size is in the row, next to the score**: `beta` (GEMMA's effect per maternal-allele copy) with `se`, and `additive` = `−beta/2`, which is the number GeneNetwork's own tables and CSV show. Quote `additive` with its sign ("each maternal-allele copy lowers the trait by X" / the other way) and remember it is *this run* on these samples, not the `additive` in trait info, which came from the published mapping. A note also carries GEMMA's own variance component for the null model (`-pve`): under LOCO there is one per chromosome, and the note lists **every** one — `chr1 0.6674±0.3451, chr2 0.8426±0.3668, …` — with the range and the two extremes (plus their `vg`/`ve`) spelled out. Quote per-chromosome values, not an average: they are twenty different kinship matrices. `pve` is the share of variance the kinship explains — the same quantity as `/compute/bulklmm`'s `h2`, per chromosome instead of genome-wide — so it is the honest way to check the two engines against each other. It is scale-free: compare `pve` with `h2`, never `vg` with `σ²_e`. A `pve` of exactly 0 or 1 is a boundary (the standard error then reads "not available"), a direction rather than a proportion. * **Choosing individuals**: every group-mapping endpoint (`/compute/gemma`, `/compute/gemma/ravanan`, `/compute/rqtl`, `/compute/qtl2`, `/compute/fitqtl`, `/compute/bulklmm`) excludes parents and F1s by default. `include_parents_f1: true` keeps them all; `individuals: {include: [names], exclude: [names]}` selects by name, `exclude` winning, with an unknown name a 422 that lists the panel. Remember *including them changes the kinship, not the fit* — GEMMA and R/qtl compute kinship without phenotypes, so parents/F1s shape relatedness while the association uses only animals with values; the reply's Genotype note gives both counts. * `/compute/bnw`: a Bayesian network over a set of traits (BNW). Give `group` + `traits`, or `values` (one entry per variable with per-sample values). Columns `from, to, posterior, in_structure`; one row per candidate edge, strongest first. **A posterior is not a p-value** — it is the share of the k-best networks containing the edge, i.e. how often the search drew that arrow in this sample, so quote it as a structural claim and never as a threshold on a QTL. Parameters default to BNW's own (maxparents 4, k 1, tiers 3, threshold 0.5) and the reply names the ones used. At most 40 variables, at least 3 samples complete across every variable. `POST /plot/bnw` draws it from the posted reply — the network by default, `?kind=posterior` for the matrix of every candidate edge (arrow width and label are the posterior; dotted lines are edges that did not clear the threshold). `POST /compute/bnw/features` turns the network into traits: each node with parents becomes a module (node + parents) scored by the first principal component of its standardised members; `?module=NAME` returns that module's `{name, value}` scores, which go straight into `/compute/rqtl` or `/compute/gemma` as `samples`. `GET /compute/bnw/examples` lists BNW's own worked examples, and `/compute/bnw/examples/{name}` returns one as a matrix you can post to `/compute/bnw` — use them to try a real network before inventing data. * `/compute/fitqtl`: **how big** an effect is, not just where. Give it the marker names a scan found (its peaks) and it fits R/qtl's `fitqtl` at each: columns `marker, chr, pos, lod, pct_var, additive, se, t, p`. Use it after `/compute/rqtl` whenever the user asks "how much does this QTL matter" — `additive` is half the difference between the two homozygote classes (the same convention GeneNetwork's published additive uses), `pct_var` the variance explained, and `p` the model's p-value. At most 100 markers per call; markers that cannot be fitted are absent, not zero. **It fits covariates too** — `covariates: [{"case_attribute": "Sex"}]`, the same shape and the same coding as `/compute/rqtl` and `/compute/gemma` — so the `additive` it reports can be the effect *after* conditioning on a confounder, which is usually the number worth quoting. The reply names any covariate column it had to leave out because it does not vary among the phenotyped individuals (R/qtl would otherwise stop with "system is exactly singular"). Without covariates its `lod` at a marker equals `/compute/rqtl`'s LOD at that marker; with one fitted it is a different model, so the two no longer match — quote the scan for "where", this for "how much". * `/compute/qtl2` with `"rqtl2_cross": "do"`: a **multiparent** scan — 8 founder alleles, an LMM with LOCO kinship. Only a cross that ships founder genotypes can do this (a `.geno` is a biallelic collapse); `do` is the one this deployment has. Pass `samples` by name; the LOD means "some founder differs", not "this strain differs" — per-founder effects are not returned yet. Minutes of genotype probabilities the first time, cached per cross. * `/compute/qtl2`: the same panel through **R/qtl2** — genotype probabilities and an LMM with `loco` kinship (`"kinship": "none"` turns it into a Haley-Knott scan on probabilities). Rows are `marker, chr, pos, lod`, so `/plot/rqtl` draws them unchanged. Reach for it when a scan's peaks depend on relatedness, or when the panel is multiparent; it is slower than `/compute/rqtl` (a DO cross costs minutes of genotype probabilities, cached per panel). Autosomes only, and `notes` says what was left out. * **Need a GEMMA threshold?** This API refuses `nperm` on GEMMA (one LOCO scan per permutation, one GEMMA slot). Two routes: `/compute/bulklmm` with `nperm` (rotation null, seconds), or run GEMMA's own `--permutate` locally on the files the reply names (exact GEMMA null, hours). R/qtl's `nperm` is a sanity check only — its null ignores relatedness. * **Need a threshold, not just a LOD?** `POST /compute/bulklmm` (BulkLMM.jl) fits an LMM per marker with a global kinship matrix and permutes by rotation, so `"nperm": 200` is cheap and its thresholds belong to the model that made the LODs (`notes` list alpha 0.10/0.05/0.01 and the seed). It is **not** GEMMA's LOCO model — say so when quoting a p-value — and `h2_null` near zero means it collapsed to a fixed-effects fit. One trait at a time is **serial** in Julia (the package threads only its many-trait scan); blocks of markers are scanned in parallel by the driver, so a reply's `notes` say how many ran. * **Genotypes at a peak** (the only genotypes the API returns): `GET /groups/{g}/traits/{id}/peak-genotypes?marker=` re-runs the scan, confirms the marker is at the peak (within `?window=` Mb, LOD ≥ `?min_lod=`) and returns one call per animal **that has a value** — `422 not_the_peak` / `not_a_hit` otherwise. Plot it by posting that reply's `genotypes` with the trait's values to `POST /plot/genotypes` (R/qtl book Fig 4.1: genotype class across, value up). * `.../markers/{marker}/classes?trait=` summarises a trait's values per genotype class (`n`, `mean`, `sd`, `min`, `max`) at one marker — how you see that a peak rests on two animals, or that a marker is nearly monomorphic after a conversion drops `H`. Genotypes never leave the API. * `/genes/{symbol}/aliases` returns Wikidata's alias list for a gene (cached seven days in `gene-aliases.lmdb`; `?species=human` for the ortholog's labels). A `"stale": true` answer means Wikidata was unreachable and this is the last value -- not that the gene has no aliases. * `/probesets/{name}` finds every dataset carrying a probeset id (indexed; suffixed `_A` variants included, `name` says which matched) and `/genes/{symbol}/traits` lists a symbol's expression traits across datasets with a real total, `?sort=lod`, `?min_lod=`, `?group=`. Neither returns values — a token is enough for the metadata, but values stay per-dataset: follow `next` to `/datasets/{ds}/probesets/{name}/values`. * `/groups/{group}/genotypes/{version}/markers` lists marker **names** only; `.../markers/{marker}/location` gives one marker's `name`/`chr`/`cM`/`Mb`. Use `?chr=&page=&limit=`. * `/groups/{group}/genotypes/{version}/frequencies` (and `.../markers/{marker}/frequencies` for one locus; for a file no group lists, `/groups/unlisted/genotypes/{name}/frequencies`): how the panel is actually called — `{"B": 0.49, "D": 0.49, "H": 0.02}`. Use it before trusting a flat scan: a marker that is 99 % missing produces the same flat LOD as real data. Probability-valued files are binned 0.05 wide (`?bin=` to change); a file over 256 MB is refused (413) rather than read. * Compute replies carry the class-count hint at their own peak (`.../markers/{peak}/classes?trait={trait_id}`): filling in the trait id is how you check a peak before reporting it, and `/compute/qtl2`'s `notes` explains what a `riself` conversion did to `H`. * **Uploading new traits/covariates**: `POST /matrices` with `{"name","genotype_version","contents":"id,trait1,cov1\nBXD1,1.2,3\n…", "kinds":{"cov1":"covariate"}}` — a delimited table whose first column names the individuals. It is checked against that genotype version's panel (same individuals, each named once) and refused with the difference otherwise. Each column becomes a basket `.`: map it with `{"basket": "."}` on any compute endpoint. `GET /matrices` lists yours, `GET /matrices/{name}/values` reads the table back. `scripts/example-matrix.py` writes a valid example from a genotype file. * **Uploading genotypes**: `POST /uploads/genotypes` with `{"name":"-GENONAME","base_version":"BXD.4","contents":""}` — tab-delimited, header = individuals from column 4, then marker/chr/pos. It **updates** a population: ≤50 new individuals, at least one overlapping, ≤200 000 markers, and the values must be the same kind as the base's (letters, whole numbers, probabilities). The reply's Every compute endpoint takes it by name — `{"genotype_upload": ""}` on `/compute/{rqtl,gemma,qtl2,fitqtl}` — with `samples` (or a `basket`) matched **by name**; the server writes the form each engine reads. Uploading the same name again **replaces** it, and the digest in the reply says which version ran. A one-chromosome upload scans flat or empty under LOCO kinship — use `"kinship": "overall"` if that is all you have. `scripts/example-genotype-upload.py` makes a valid one from any `.geno`. * **Changing values**: there is no write endpoint for traits, and there never will be — but a user's own column can be stored and mapped. `POST /baskets` with `{"name", "values":[{name,value}…]}` (values may be numbers, strings like `"NA"`, or null), `PATCH /baskets/{name}` to edit (a new revision, a new digest), then `POST /compute/qtl2` with `"basket": ""` instead of `samples`. Baskets are private to the token's user, live in their own SQLite file, and a mapping reply records the revision and digest it used — quote them when reporting a result. Each edit is a new revision with its own timestamp (`GET /baskets/{name}/revisions`), old revisions stay readable (`?revision=N`), and `"basket": "name@2"` maps a pinned one — quote `name@revision` when a result must be reproducible. All four compute endpoints take `basket`. Baskets live in one file per user under `BASKET_DIR`, named for the user, so a new token still finds them. `GET /baskets` lists them, `DELETE` removes one. * Follow `next` freely: every link is a `GET` that returns 200, and the API offers the whole graph from any response (genotypes -> frequencies -> markers -> a marker's location/tally -> PheWAS; trait -> values -> genotypes; `/version` -> every listing). `actions` holds the `POST`s, which are not linkable. * `/plot/rqtl`, `/plot/gemma`: post the compute reply back, receive an `image/svg+xml` plot. Do not draw your own. * Options (all optional, add to the posted body; full list in `components.schemas.RqtlRequest` / `GemmaRequest`): * R/qtl: `nperm` (permutations; **100 takes about 14 s** and adds `significant`/`suggestive` thresholds that `/plot/rqtl` then draws), `method` (`hk` default, `ehk`, `mr`, `em`, `imp`), `model` (`normal` default, `np`), `control` (a marker to condition on), `genotype_version`, `include_parents_f1`, `covariates`. * GEMMA: `lmm` (default 9), `genotype_version`, `include_parents_f1`, `covariates`, `trait_name` (`"BXD::20010"` — optional, but it is what the reply reports and what `/plot/gemma` titles the picture with; without it the title is `::gemma`). `nperm` is refused (`422 permutations_not_supported`): for significance send the same values to `/compute/rqtl` with `nperm`. * **Normalizing before mapping** (both engines, same three fields): `normalize` = `none` | `log2` | `log10` | `sqrt` | `negate` | `zscore` | `robust_zscore` | `rank` (GN2's "Quantile"/`qnorm`; note it is a rank-based inverse-normal transform, not array quantile normalisation), plus `winsor_quantile` (e.g. `0.05` for the 5th/95th percentile) or `winsor_fences` (`1.5` = GN2's outlier rule, `Q1-1.5*IQR`). They run transform → winsorise → standardise, and the reply's `notes` name the pipeline and any clipped/​outlying values. `GET …/values?normalize=` does the same for the values endpoint and carries the fields into the suggested mapping bodies, so use that when you want the fetched numbers to be the mapped ones. `zscore` is the population sd, matching genenetwork.org; effects (`beta`/`additive`) come out on the transformed scale, so say which scale you are quoting. * **Covariates** (`/compute/rqtl`, `/compute/gemma` **and `/compute/fitqtl`**): `covariates: [{"case_attribute": ""}]`. Worth trying when the study's own metadata could confound the trait — sex, age at death, batch/epoch, cohort, sequencing depth — because they are fitted alongside the marker, and a peak that survives them is the stronger result. **Read `.data.descriptions` before choosing one**, not just `.data.names`: the names do not explain themselves, and the descriptions do (BXD: `SeqCvge` = "Depth of DNA sequencing for 1 male of each strain in 2016", `Epoch` = BXD family subgroups, `Epoch1(BXD1-32)`, `Epoch2-6 (BXD33-220)`, `Status` = `Live= Available at JAX, Cryo=Cryopreserved only, Extinct`, `RRID` = a SciCrunch identifier). BXD has no `Sex`; an unknown name is `422 covariate_unavailable` listing the ones the group does have. * **Then read the covariate line in `notes`.** It reports the coding and the complete cases, e.g. `Covariate Status (Live=…, Cryo=…): categorical coded Cryo=0, Cyro=1, Ext=2, Live=3, UTHSC=4, UTHSC2=5, 90 of 93 samples have a value, 34 with both the trait and the covariate; 6 levels`. Report that *n* next to any LOD/p change — a peak that shifts because 60 samples left the fit is not a discovery. Note the data's own typo: `Cyro` and `Cryo` are two levels. * **Categorical vs numeric.** Numeric text passes through as numbers; anything else with 2–20 distinct values among *the strains being mapped* is coded 0..k−1 in sorted order; more than 20, a single level, or a `map` that misses a value is a `422` asking for an explicit `map`. Measured on BXD (trait 10001, `BXD.4`): `RRID` is refused — `422 … 82 distinct values, too many to code automatically` — while `Status` (6 levels) and `Epoch` are accepted. * **Compound values are a trap.** `SeqCvge` mixes numbers with two-library values (`40+31L`) and carries 30 distinct values across the group: if few enough fall inside the mapped panel it is accepted, but as a *categorical* — and the GEMMA run then fails (`502 gemma_failed`) rather than telling you why. The same is true of any mostly-numeric attribute with a few odd entries. Send the coding yourself: `{"case_attribute": "SeqCvge", "map": {"40+31L": 71, "40": 40, …}}`, or inline `{"name": …, "kind": "numerical", "values": […]}`. * **GEMMA** gets k−1 dummy columns (first level = reference) and *drops* a level none of the fitted individuals carries; the reply says so (`GEMMA fits 4 covariate column(s): Status.Ext, …` and `Covariate column(s) left out because they do not vary …`). R/qtl takes the single level column and dummy-expands it itself, so its note has no such line. * Missing covariate values drop those individuals from the fit (both engines), which `notes` reports as `Missing covariate values: …`. * At most 8 covariates, and they need a `group` (the individuals are aligned by name, which a bare `genotype_file` request does not carry). * `genotype_version`: list them with `/groups/{g}/genotypes` (BXD.8 is current). * Timings seen: R/qtl about 7 s (1 MB reply), GEMMA about 3 s (1.9 MB), plots instant. They are synchronous; `?id=` only adds progress logging. * Progress: `GET /compute/{rqtl,gemma}/status/{id}` returns `lines` and a `pointer` to pass back as `peek`; the last line is `COMPLETION_MARKER` or `FAILURE_MARKER: `. It exists only for an `id` you passed on the POST (otherwise `404 run_not_found`). * `/plot/*` accepts the whole compute reply or just its `data`; query params `trait`, `title`, and `bonferroni=false` (GEMMA only). The title names the trait from the reply's own `Trait: ...` note (both compute endpoints write it first), so posting the whole reply back is enough; `?trait=` wins over the note. * Two traits really do draw two different plots (traces, peaks, annotation); the title is the one thing that was identical before the note existed. * `/compute/correlation` is a stub with canned data. Do not use it. * You never supply genotypes: the server uses its own files. **PheWAS (GET)** `/groups/{group}/phewas?locus=` or `?chr=&mb=`, `window` (bp, default 10000), `genotype_version`. Returns published traits whose recorded peak marker lies in the window, strongest first. An empty list is a real answer. **Data (GET, all need the token)** | path | gives you | |---|---| | `/species`, `/species/{s}`, `/species/{s}/datasets` | species, expression datasets | | `/groups?species=`, `/groups/{g}`, `/groups/{g}/strains`, `/groups/{g}/datasets` | populations and samples | | `/groups/{g}/case-attributes` | the group's metadata: `.data.names`, and `.data.descriptions` **saying what each one means** — read those before fitting a covariate | | `/groups/{g}/traits`, `/groups/{g}/traits/{id}`, `/groups/{g}/traits/{id}/values` | published traits (the first is the bulk list) | | `/groups/{g}/traits/{id}/case-attributes/{attribute}` | a trait's case-attribute values, in the trait's own sample order, plus `description` and a `distinct` count — the honest way to see whether an attribute is usable as a covariate (a constant one is not) | | `/groups/{g}/genotypes[/{version}[/markers]]` | genotype versions and markers, and the marker names `/compute/fitqtl` takes | | `POST /compute/{rqtl,gemma,fitqtl}` | mapping (POST, not a GET): the scan, the LMM, and effect sizes at named markers — all three take `covariates` and `normalize` | | `/datasets/{ds}[/probesets[/{p}[/values]]]`, `/traits/{ds}[/{t}[/values]]` | expression data | | `/publications/{pubmed_id}/traits` | traits a paper reports (PubMed id, not `publication_id`) | | `/accessions/{acc}`, `/mapping-methods` | GN accession lookup, methods offered | **Bulk jobs.** Fetch `/groups/{g}/traits` once (the whole PublishXRef list: `id`, `phenotype_id`, `dataset`, `abbreviation`, `nsamples`; about 15 000 for BXD), then `.../traits/{id}/values` per trait, a few at a time. Prefer one bulk call to a thousand small ones; do not use `/search` for bulk work. **Other examples:** `/datasets/Br_U_0803_M/probesets/100001_at` (one expression trait), `/publications/2760876/traits` (PubMed id), and `/search?q=cerebellum AND group:BXD&limit=5`. ## 3a. Gene link-outs: where else to send a user about a gene GeneNetwork2's trait page links out to a dozen services by gene symbol. These are the ones that still resolve (checked 2026-11-11, HTTP 200 and — where the page is not a JavaScript shell — the gene actually found), with the URL template to hand to a user. `{SYMBOL}` is the gene symbol; taxa are **mouse 10090, rat 10116, human 9606**. | service | URL template | what it is for | |---|---|---| | **RGD** (gene search) | `https://rgd.mcw.edu/rgdweb/elasticResults.html?term={SYMBOL}&category=Gene&species={Mouse\|Rat\|Human}` | the best single hub *for mouse and rat*: orthologs, phenotypes, expression, QTLs. Works for all three species | | **MGI** | `https://www.informatics.jax.org/searchtool/SearchResults?query={SYMBOL}` | mouse gene authority: alleles, phenotypes, knockouts | | **Alliance of Genome Resources** | `https://www.alliancegenome.org/search?category=gene&q={SYMBOL}` | cross-species reference: orthologs and disease models | | **NCBI Gene** | `https://www.ncbi.nlm.nih.gov/gene/?term={SYMBOL}[gene]+AND+mouse[orgn]` | the canonical record; also the source of Gene IDs | | **Wikidata** | `https://www.wikidata.org/wiki/{QID}` | identifiers and aliases across databases. The QID is what `/genes/{SYMBOL}/aliases` returns, so this link costs nothing to build | | **UniProt** | `https://www.uniprot.org/uniprotkb?query=gene:{SYMBOL}+AND+organism_id:{TAXON}` (or `…/uniprotkb/{ACCESSION}`) | protein entry, domains, variants. Search by symbol, link by accession | | **Ensembl** | `https://www.ensembl.org/Mus_musculus/Gene/Summary?g={ENSEMBL_ID}` | genome browser, transcripts, orthologs. Needs the Ensembl gene id | | **GTEx** | `https://www.gtexportal.org/home/gene/{HUMAN_SYMBOL}` | human tissue expression. **Human symbol only** — a mouse symbol shows an empty report | | **STRING** | `https://string-db.org/cgi/network?identifier={SYMBOL}&species={TAXON}` | protein–protein interaction network | | **GeneMANIA** | `https://genemania.org/search/{mouse\|rat\|human}/{SYMBOL}` | functional association network, per species | | **Open Targets** | `https://platform.opentargets.org/search?q={SYMBOL}` | target–disease associations (human) | | **Human Protein Atlas** | `https://www.proteinatlas.org/search/{SYMBOL}` | human tissue/cell expression at protein level | | **EBI GWAS Catalog** | `https://www.ebi.ac.uk/gwas/search?query={SYMBOL}` | published GWAS associations | | **BioGPS** | `http://biogps.org/?org=10090#goto=genereport&id={SYMBOL}` | mouse expression across tissues and datasets | | **Allen Brain Atlas** | `https://mouse.brain-map.org/search/show?search_type=gene&search_term={SYMBOL}` | in-situ hybridisation in mouse brain | | **GeneCup** | `https://genecup.org/progress?type=GWAS&query={SYMBOL}` | literature-mined gene–disease links | **Do not offer these** (checked at the same time): HomoloGene (NCBI retired it in 2021 — the URL answers with a notice), UniGene (retired, HTTP 410), Genotation and GeneBridge (no DNS/connection), PhenoGen (404), the old chibi GEMMA gene page, wiki-Pi. PANTHER answers 403 to a bot but works in a browser, so offer it only as a "works in your browser" link: `https://www.pantherdb.org/genes/geneList.do?searchType=basic&fieldName=all&organism=all&listType=1&fieldValue={SYMBOL}`. **How to use them.** Quote the symbol and the species you found it in, and let the user click. Do not synthesise what the other service says — say *"RGD has a gene report for Shh (rat)"* and give the link, not a paraphrase of a page you did not read. For a **rat** symbol prefer RGD and GeneMANIA(rat); for a **mouse** symbol all of the above except Human Protein Atlas and GWAS apply; for a **human** symbol use GTEx, Open Targets and the Protein Atlas. **The API builds them for you.** `GET /genes/{SYMBOL}/aliases` returns a `links` array (`[{name, href, description}]`) built from the table above for the symbol and species it actually matched -- RGD, MGI, BioGPS and the Allen Brain Atlas for a mouse gene; RGD, GeneMANIA and UniProt for a rat one. It also returns `label`, the entity's canonical symbol in that species, which is how you get the **human** symbol a human-only service needs: GET /genes/Shh/aliases?species=human → label "SHH", links […] including GTEx → https://www.gtexportal.org/home/gene/SHH So a mouse `Shh` reaches GTEx through the human ortholog's label, and you never hold a URL template yourself. `stale: true` means Wikidata was unreachable and the cached entity was served -- the links still work, the label may be absent from an older cache entry (retry and it refreshes). ## 4. Gotchas that cost time * Compute replies are big (about 21 000 rows; GEMMA about 2 MB). Save to a file, inspect with `jq` / `python3 -I`. Piping to `head` forces a rerun. * Rows are positional; names are in `meta.columns`. * Not every value maps: trait 20010's 41 values map as 39, because C57BL/6J and DBA/2J are not in the BXD.8 genotype file, and `include_parents_f1: true` does not change that. `/groups/{g}/strains` flags such samples with `used_for_mapping: false`, which predicts it. `notes` on the reply says how many matched; tell the user. * A peak is often a plateau: neighbouring markers share the same LOD / p-value (trait 20010's GEMMA maximum, 5.79, is shared by several markers from 182.6 to 184.3 Mb). So the trait record's `locus`, R/qtl's peak, the plot summary and a reply's `next` can each name a different marker of the same strength. That is not an error: say "peak region" and use the marker the reply's `next` gives. * **Before you believe a peak, count the animals in its genotype class.** `GET /groups/{g}/genotypes/{v}/markers/{marker}/classes?trait=` gives `n`, `mean`, `sd`, `min`, `max` per class at that marker (no genotypes, no per-animal values). A peak carried by 1–2 animals is a different claim from one carried by a hundred, and the reply says which classes are thin. On DOL trait 10003 GEMMA's top marker `JAX00418132` (chr16 36.35 Mb, −log10 p 7.15) is `A n=2 mean 1.8`, `H n=20 mean 2.53`, `B n=171 mean 3.95`: the two A animals are the trait's two lowest values, and GEMMA's next two hits are the same pattern mirrored (`1 A` among 171, then `2 B` among 171). That block is a haplotype carried by a handful of extreme animals, not a population effect — say so rather than reporting 7.15 as a locus. * **When R/qtl finds a peak R/qtl2 doesn't**, the model is the first suspect, not the data: R/qtl fits each marker's own calls (dropping the least frequent of three codes), R/qtl2 fits HMM probabilities with the H calls re-imputed. `{"genotype_model": "f2"}` keeps the het calls as a third genotype and shows what that recovers (BXD trait 16627: LOD 3.73 → 9.68 at `rs33299937`, against R/qtl's 9.59) — a diagnostic, not a result, since an inbred line has no heterozygotes. The reply says `Cross: riself` or `Cross: f2`: quote it. * **`H` is dropped for R/qtl2 on a `riself` panel** (`@type:riset`: BXD, DOL, most RI sets), and kept by GEMMA as dosage 0.5. R/qtl2 writes those cells as missing and its HMM re-imputes them from flanking markers, so the animals are not lost — the observation is. A marker that is mostly `H` becomes nearly monomorphic for R/qtl2 (2 A vs 171 B at the marker above) and cannot produce a peak there, which is why the two engines can name different markers of the same region. Check `/frequencies` or `/classes` before calling one engine wrong. * **GEMMA's `af` is not an allele frequency.** It is `mean(dosage)/2` under the BIMBAM coding the deployment ships (B=1, H=0.5, A=0), so `af 0.469` for `JAX00418132` is half the B-allele frequency 0.935, not the A-allele frequency 0.065. Take allele frequencies from `.../markers/{marker}/frequencies`. Likewise `allele1`/`allele0` in GEMMA's output are the BIMBAM `X, Y` placeholders, so the sign of `beta` is not a strain. * **Do not compare scores across engines as if they were the same scale.** GEMMA reports −log10 p, R/qtl and R/qtl2 report LOD, qtlreaper reports LRS (`lod = lrs / 4.604`). The same region at "7.15" and "3.92" is not a disagreement, and an LMM's LOD is not a Haley-Knott LOD. * PheWAS window matters: at the trait-20010 peak, 10 kb finds 4 traits and 1 Mb finds 19. The reply's `next` offers a ten-times wider window. * Public-data rule: `public > 0` and `confidentiality < 1`, or older than five years; otherwise 403, do not retry. Metadata is visible even when values are not. * There are no write endpoints, and the search index holds public documents only. * Older docs said `POST /compute/*` is refused with 403. On this build it works. If you do get a 403 there, that is the answer: do not work around it. If the task needs private data or computation you are not granted, stop and ask a human: that needs an account and a grant, not a different URL. ## 5. Search `/search?q=` is Xapian: juxtaposed terms are **OR**, so write `AND`; free text is stemmed and case-insensitive; quoted phrases, wildcards and hyphenated words match nothing. Fields: exact `group`, `dataset`, `species`, `author`, `tissue`, `symbol`, `chr`, `peakchr`, `position`; stemmed `description`, `rif`, `wiki`; ranges `mean`, `peak`, `peakmb`, `additive`, `year` (`field:lo..hi`). Aliases: `pos` for `position`, `RIF` for `rif`, `WIKI` for `wiki`; `Hs`/`Mm` are species-location shorthands. Values are lowercased, so case does not matter. A prefix whose value is not in the index matches nothing; that is "no such value", not an error. `NEAR`/`ADJ` do not work either. A dangling operator or unbalanced quote is `422 invalid_query` with Xapian's reason in `detail`: fix it, do not retry. `type=phenotype|gene`, `limit` up to 100, `page`. There is no total: counting means paging, and each page costs time proportional to the matches left, so do not page through thousands of hits. For bulk work use `/groups/{g}/traits`. `.data` is an array of hits. Each has `name` (the **trait id**, e.g. `"21614"`), `dataset`, `dataset_fullname`, `group`, `species`, `authors`, `year`, `description`, `mean` and more. Without `group:` you also get other groups' phenotypes (for example `C57BL-6JxC57BL-6NJF2Publish`), whose ids only mean something inside their own group; add `AND group:BXD` to stay in BXD. ## 6. Tokens and errors A GeneNetwork team member issues the token by hand; it cannot be self-served. If you have none, ask: "To read GeneNetwork data I need a GeneNetwork API token (a JWT) for https://vesuvio.pu.ac.ke/gn4. It is issued by hand; it helps to say who it is for, which groups/datasets, how long (default 30 days) and what for." Ask the user to pass it out of band, not in a public issue, a commit or a shared chat. Send it as `Authorization: Bearer ` on every request except `/version` and `/openapi.json`, and keep it for the whole session. Never put it in a URL, a commit, a log line or anything shown to someone else, and do not hand it to other users or paste it into shared state. | status | do | |---|---| | 401 `invalid_token` | `detail` says `ExpiredSignature` (expired) or `InvalidIssuer` (wrong deployment). You cannot renew it: tell the user, ask for a fresh one quoting the `request_id`, retry **once**, then stop and report; do not loop or look for another credential | | 403 `forbidden` | not public: do not retry | | 404 `not_found` | check the name; `next` lists valid values | | 422 `invalid_query`, `invalid_lmm` | fix the query or parameter; do not retry | | 429 | back off and honour `Retry-After` (not emitted today, but in the contract) | | 502/503 | transient: back off, max about 5 tries | Errors are RFC 7807-style JSON (`type`, `title`, `status`, `code`, `detail`, `operation_id`, `spec_url`, `request_id`, sometimes `next`); `type`, `spec_url` and `next` are absolute paths under `/gn4/`. Quote `request_id` when reporting a problem (also in the `x-request-id` header). No rate limit is configured, but be considerate: a few concurrent requests, no tight polling loops. ## 7. Keep this skill current: reload it if GeneNetwork changes mid-session Do not rely on a version written into this file. Instead, **remember the build you saw at the start of the session** and compare against it later. 1. At the start (section 0), note `curl -s https://vesuvio.pu.ac.ke/gn4/version | jq -r .data.git` as your session baseline. 2. Check `/version` again when something behaves differently from what is written here (an unexpected 404, a new field in `actions`, a path in `openapi.json` this file does not mention), and before a long or repeated piece of work. 3. If the revision is no longer the baseline, GeneNetwork was updated while you worked. **Suggest to the user that you reload SKILL.md from the repo** (`docs/www/SKILL.md` in `https://git.genenetwork.org/genenetwork4`; a sparse clone of `docs` is enough), and use that copy from then on. One line is enough, for example: "GeneNetwork was updated since we started ( -> ); I'd reload SKILL.md from docs/www/ to pick up any changes. Shall I?" Then set the new revision as your baseline. 4. If the repo is unreachable from the sandbox, fall back to `/gn4/openapi.json`, which always matches the running server, and tell the user the skill file itself could not be refreshed. ## 7a. Project memory: the answers this repo has already worked out This repository carries an `ai-memory` store (`ai-memory.sqlite`) linking each commit to the reasoning behind it, plus an optional Xapian topic index over both. A previous session's context window is gone; this is what survives it, and it is cheaper than re-deriving anything: ```sh ai-memory -C genenetwork4 search 'pve' # SQLite FTS over the reasoning: ~15 ms ai-memory -C genenetwork4 why gemma.compute_gemma # commits touching a symbol, with its reasoning ai-memory -C genenetwork4 --index DIR search 'topic:pve repo:genenetwork4' # topic index: ~100 ms ``` `topic:` terms come from the session id, so `topic:ravanan-slurm`, `topic:reference-genomes`, `topic:gemma-pve` are the entry points that exist. Two things to look for before starting work that touches an engine, a cache or a cluster: - **a RUNBOOK entry** — the operational ones carry the actual commands, ports and fingerprints, e.g. bringing up the Slurm container from `gn-machines` and testing `/compute/gemma/ravanan` against it, with the four prerequisites that were needed the first time (a ravanan build with the `--slurm-api-endpoint` fix, `ccwl` on `PATH`, the server's `HOME` writable inside the container, `GUIX_PACKAGE_PATH` for the manifest); - **a "why is this like this" entry** — several endpoints here exist because something else was wrong (`-pve` reporting, the `next`-link graph, the reference-genome linkage). Reading it is faster than deciding they are odd. The store is per repository and local (gitignored): it survives a cleared session, not a lost machine, which is why the durable version of every conclusion is also in a commit message and in these docs. ## 8. Reference docs The docs live in the public repo and are the authoritative version: ```sh git clone --depth 1 --sparse https://git.genenetwork.org/genenetwork4 cd genenetwork4 && git sparse-checkout set docs ``` About 2.5 MB; drop `--sparse` for the whole tree. Then read: | file | what it is | |---|---| | `docs/www/SKILL.md` | this page | | `docs/search-engines.md` | how `/search` works, engines, quirks | | `README.md` | every endpoint, parameter and response shape | | `docs/nginx.md`, `docs/www/operator-checklist.md` | operator-level; you do not need them | **Clone; do not fetch the web UI.** `git.genenetwork.org/.../tree/...` pages sit behind a proof-of-work challenge (Anubis) and the raw endpoint answers 403, so a plain fetch returns an HTML challenge page, not Markdown. If a fetch comes back as HTML mentioning Anubis, stop, do not retry, and use the copy you have. The git protocol itself passes. Sandboxes sometimes cannot reach it either; `/gn4/openapi.json` is the fallback. To compare revisions, `git -C genenetwork4 log -1 --format=%h` against `.data.git` from `/version`: they need not match exactly (docs can be newer than the running build), but a large gap means the server may lack what the docs describe.