about summary refs log tree commit diff

name: genenetwork description: Use when the user wants anything from GeneNetwork (genenetwork.org): BXD and other mouse/rat genetic reference populations, traits ("get trait 20010"), per-strain values, QTL mapping with R/qtl or GEMMA, LOD/Manhattan plots, PheWAS, genotypes, or searching published phenotypes. Talks to GeneNetwork through its HTTP interface at https://vesuvio.pu.ac.ke/gn4.


GeneNetwork

The user is talking to GeneNetwork, and you are their way in. Talk about GeneNetwork in their terms: traits, groups, strains, mapping, plots, PheWAS. Do not talk about "the API", endpoints or HTTP unless they ask. Everything below is how you do it; what you say should be what GeneNetwork did.

What the user can do in GeneNetwork

When the request is open ("what can I do?", or you finish a step), offer the next natural option from this list:

Ask for What GeneNetwork gives Section
a trait ("get trait 20010") metadata (description, peak locus, LOD, n, source) and the per-strain values 3
a QTL map R/qtl scan (LOD) or GEMMA LMM with LOCO kinship (Manhattan), run on the server 3
a plot an SVG LOD / Manhattan plot, drawn by GeneNetwork 3
PheWAS every published trait of the group peaking near a position (e.g. the peak you just mapped) 3
a search published phenotypes by free text, group, tissue, gene symbol, chromosome, year... 5
populations and data species, groups (BXD...), strains, case attributes, genotype versions, expression datasets and probesets, publications by PubMed id, GN accessions 4

A natural session is: trait, values, R/qtl, plot, GEMMA, plot, PheWAS at the peak — and /compute/fitqtl at that peak when the user asks how much it matters (with a covariate, if one could confound it). Whenever the group records metadata that could confound the trait (sex, age, epoch, batch, sequencing depth), the same mappings again with that covariate fitted, to see the peak survive it. After each step, suggest the next one.

0. Do this first

  1. Check you can reach GeneNetwork (no token needed): curl -s https://vesuvio.pu.ac.ke/gn4/version and, for the docs repo, git ls-remote https://git.genenetwork.org/genenetwork4 HEAD. A sandbox proxy blocks any host that is not on the person's allowlist (connect_rejected, or CONNECT tunnel failed, 403 from git). Ask them to allow both domains in one go, not one at a time:

To work with GeneNetwork I need two domains allowed for this session: vesuvio.pu.ac.ke (the data and mapping service) and git.genenetwork.org (the docs, so I can reload this skill when it changes). Please add them under claude.ai > Settings > Capabilities (admins: Organization settings > Capabilities), then start a new message or session.

If only the repo is blocked, GeneNetwork itself still works: carry on, and tell the user the skill file cannot be refreshed (section 7). Say all this once and stop probing other routes. 2. Token. Look for one already given to you in the task context. If there is none, stop and ask (section 6). Never guess, invent, reuse a token that was not given to you, or look for another way in: POST /gn4/get-test-token is a deliberate 404, and there is no sign-up or login. A token carries no rights of its own; what you may read is decided per dataset. 3. Note the build (section 7): remember .data.git from /version as your baseline; if it changes later in the session, suggest reloading docs/www/SKILL.md. 4. Read the spec, not only this page. GET /gn4/openapi.json lists every operation and is what the server really runs; this page can lag. Use jq -r '.paths | keys[]' before saying something is not available. The descriptions inside it are the real reference: every request option is documented in components.schemas (for example jq '.components.schemas.RqtlRequest.properties' openapi.json). 5. Assume, don't ask. A bare numeric trait id (20010) is a published phenotype of group BXD (dataset BXDPublish). Try it; ask only on 404.

Base URL for everything: https://vesuvio.pu.ac.ke/gn4. /gn4 without the trailing slash redirects there; anything outside /gn4/ is a 404 from the front end, not from GeneNetwork. Only /version and /openapi.json need no token; everything else answers 401 (no token) or 404 (not routed).

1. Read the replies: GeneNetwork tells you what to do next

Every JSON reply is {"data","meta","notes","next"} and sometimes "actions".

  • notes: read them. They say how many strains were matched, what was run, and caveats.
  • next: GET links to related things. Each href already includes /gn4: resolve against the origin (https://vesuvio.pu.ac.ke$href).
  • actions: ready-to-send POSTs. On a trait's /values they are the R/qtl and GEMMA calls with the body already filled in; on a compute reply they are the matching plot call (href with ?trait= already set; post the reply back to it). Use them instead of building calls by hand. Select by href, not position: jq '.actions[] | select(.href|endswith("/compute/rqtl")) | .body'.
  • Every compute reply also has next: a PheWAS link at that run's peak marker. Read next and actions of each reply before deciding what to do.
  • meta: state (complete/partial), took_ms, request_id, plus columns/num_rows on tabular payloads.
  • meta.state: partial means usable but incomplete; the reason is in notes. Do not read a null enrichment field as "no such value in GeneNetwork"; read it as "ask again later".
  • Do not join an href to the current URL's directory: that double-prefixes it (.../gn4/gn4/...).

Always offer the next step

End every answer with one sentence saying what to do next, built from that reply's own next/actions — never from an idea of what a mapping pipeline "usually" does. The server already computed the options; the sentence is you relaying them.

  • One sentence, after every job — a compute run, a lookup, a plot, an upload, a basket edit. Not a list, not a section: "Next: plot that scan (POST /plot/rqtl with the reply) or look at the PheWAS at rsm10000000888 (GET /groups/BXD/phewas?locus=rsm10000000888)."
  • Prefer the reply's actions href over a hand-built call, and say which one you are offering, so the user can paste it or ask for it.
  • A refusal is an answer too. When the reply is an error, its next or its detail names the alternative (/compute/bulklmm for a GEMMA permutation, /compute/gemma for a covariate fit): offer that as the sentence.
  • Do not stack suggestions. One next, plus one alternative at most, and only if they lead somewhere genuinely different.
  • The user can switch it off. "Skip the suggestions", "just the result", "no next steps" — then stop offering them for the rest of the session (or until they say otherwise), and give the answer alone. Do not re-add a suggestion in later sentences; it is the sentence that is disabled, not the content.
  • Never invent a step the API cannot do. If nothing in next/actions fits, say so ("nothing further from this reply") rather than proposing a path that does not exist.

2. Workflow: map a trait and plot it

API=https://vesuvio.pu.ac.ke/gn4; AUTH="Authorization: Bearer $TOKEN"; D=$(mktemp -d)
curl -s -H "$AUTH" $API/groups/BXD/traits/20010                       # metadata
curl -s -H "$AUTH" $API/groups/BXD/traits/20010/values > $D/v.json    # values + actions
jq '.actions[]|select(.href|endswith("/compute/rqtl"))|.body' $D/v.json > $D/body.json

# R/qtl. Save the FULL reply to a file (never pipe through head), then plot it.
curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/body.json \
  $API/compute/rqtl > $D/rqtl.json
curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/rqtl.json \
  -D $D/h.txt -o rqtl.svg "$API/plot/rqtl?trait=BXD::20010"; grep -i x-plot-summary $D/h.txt

# GEMMA: same shape. Use /compute/gemma/ravanan -- it is the default runner
# (see "Which GEMMA endpoint" below).
jq '.actions[]|select(.href|endswith("/compute/gemma"))|.body' $D/v.json > $D/gbody.json
curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/gbody.json \
  "$API/compute/gemma/ravanan?id=t20010" > $D/gemma.json
curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/gemma.json \
  -o gemma.svg "$API/plot/gemma?trait=BXD::20010"

# PheWAS at the peak: follow the compute reply's own `next` (10 kb window), or widen it
jq -r '.next[0].href' $D/rqtl.json
curl -s -H "$AUTH" --get --data-urlencode locus=<peak marker> --data window=1000000 \
  $API/groups/BXD/phewas              # window = half-width in bp

Give the user the SVGs (they render directly) and quote the x-plot-summary header (peak, and the reference lines drawn when you asked for them).

Short API tokens (gn4ai_...)

A second credential format beside gn-auth's JWT, for agents and scripts that should not carry a user's password or a long-lived JWT. Opaque: nothing is stored inside it, and everything about it lives in the server's registry (one SQLite database, tokens.sqlite).

  • Administer them from this machine only. scripts/gn4-token.py create|list|revoke|who (routes /tokens, /tokens/{id}, /tokens/who). Those routes answer only a direct caller on this machine — the server checks the TCP peer address, not a header — and a request that arrives through a proxy is refused, so this can never be driven from the network.
  • The token is shown exactly once, by create. Only its SHA-256 is stored; there is no way to recover or enumerate one. list shows id, subject, scope, status, prefix, expiry and last use.
  • Send it as Authorization: Bearer gn4ai_... — never as a query parameter, which would land in nginx's log and in next links.
  • Public data only. It works on every token-only endpoint (search, compute, plots, genotype metadata and frequencies, baskets, uploads, matrices) and on secured endpoints for public rows; a confidential dataset answers 403 telling the caller to use a gn-auth JWT, because gn-auth cannot validate an opaque token.
  • Revocation takes effect on the next request, with no restart: it is a row in the registry. Expiry is fixed when minted (30 days by default).
  • 64 characters, gn4ai_ + 56 body characters + a 2-character checksum, from an alphabet with no look-alikes (0/O, 1/l/I) — a typo is refused before the registry is asked.
  • In the log, one JSON line per request names the person and the credential: {"event":"request","ts":…,"user":"zachs","scheme":"opaque","token":"agent-1"} — never the token or its hash. scripts/gn4-log.py -f LOG renders that as a table with the handle resolved against the registry, so it shows the subject as the registry has it now and the token's status today ([revoked], [expired], [unknown]); --json enriches the records for jq, and --summary counts requests per caller and credential.

Which GEMMA endpoint: ravanan by default

Two endpoints run the same computation, and the reply shape is identical:

endpoint runs
/compute/gemma/ravanan gemma itself, one invocation per chromosome, scattered by ravanan (Slurm)
/compute/gemma gemma-wrapper (kinship, then GWA, both --loco) as a server subprocess, no cluster

Use /compute/gemma/ravanan by default. Reach for /compute/gemma only when:

  • Slurm is down or job submission fails (502/ravanan_failed, or a job that never leaves PENDING) -- the in-process runner needs no cluster;
  • the user asks for it, or the request needs something only it supports (covariates, normalization -- ravanan refuses those with a 422);
  • you are testing or verifying results. The wrapper endpoint is the reference: send the same body to both and compare rows. scripts/gemma-compare.py --base URL --body FILE does exactly that, prints both wall-clock times, and exits non-zero if any rs disagrees on beta, se, l_mle or p_lrt.

Both are warm-cached and both are cheap to repeat: a repeat of an identical request comes back from cache (ravanan's store, or gemma-wrapper's --cache-dir) in ~1-2 s against ~12-15 s for a first run, so re-running to verify costs almost nothing.

3. GeneNetwork functionality, in detail

Mapping and plots (POST, verified working)

  • /compute/rqtl: R/qtl scanone (Haley-Knott) over the group's genotypes. Columns name, chr, pos, lod. nperm=0 by default, so no permutation lines.
  • /compute/gemma: GEMMA LMM with leave-one-chromosome-out kinship. Columns chr, rs, ps, n_miss, allele1, allele0, af, beta, se, l_mle, additive, p_lrt. Pass ?id= and poll /compute/gemma/status/{id} for long runs. Per-marker p-values only; the plot draws no reference line unless asked — ?bonferroni=true for GEMMA's own, ?thresholds= for another engine's. Effect size is in the row, next to the score: beta (GEMMA's effect per maternal-allele copy) with se, and additive = −beta/2, which is the number GeneNetwork's own tables and CSV show. Quote additive with its sign ("each maternal-allele copy lowers the trait by X" / the other way) and remember it is this run on these samples, not the additive in trait info, which came from the published mapping. A note also carries GEMMA's own variance component for the null model (-pve): under LOCO there is one per chromosome, and the note lists every one — chr1 0.6674±0.3451, chr2 0.8426±0.3668, … — with the range and the two extremes (plus their vg/ve) spelled out. Quote per-chromosome values, not an average: they are twenty different kinship matrices. pve is the share of variance the kinship explains — the same quantity as /compute/bulklmm's h2, per chromosome instead of genome-wide — so it is the honest way to check the two engines against each other. It is scale-free: compare pve with h2, never vg with σ²_e. A pve of exactly 0 or 1 is a boundary (the standard error then reads "not available"), a direction rather than a proportion.
  • Choosing individuals: every group-mapping endpoint (/compute/gemma, /compute/gemma/ravanan, /compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm) excludes parents and F1s by default. include_parents_f1: true keeps them all; individuals: {include: [names], exclude: [names]} selects by name, exclude winning, with an unknown name a 422 that lists the panel. Remember including them changes the kinship, not the fit — GEMMA and R/qtl compute kinship without phenotypes, so parents/F1s shape relatedness while the association uses only animals with values; the reply's Genotype note gives both counts.
  • /compute/bnw: a Bayesian network over a set of traits (BNW). Give group + traits, or values (one entry per variable with per-sample values). Columns from, to, posterior, in_structure; one row per candidate edge, strongest first. A posterior is not a p-value — it is the share of the k-best networks containing the edge, i.e. how often the search drew that arrow in this sample, so quote it as a structural claim and never as a threshold on a QTL. Parameters default to BNW's own (maxparents 4, k 1, tiers 3, threshold 0.5) and the reply names the ones used. At most 40 variables, at least 3 samples complete across every variable. POST /plot/bnw draws it from the posted reply — the network by default, ?kind=posterior for the matrix of every candidate edge (arrow width and label are the posterior; dotted lines are edges that did not clear the threshold). POST /compute/bnw/features turns the network into traits: each node with parents becomes a module (node + parents) scored by the first principal component of its standardised members; ?module=NAME returns that module's {name, value} scores, which go straight into /compute/rqtl or /compute/gemma as samples. GET /compute/bnw/examples lists BNW's own worked examples, and /compute/bnw/examples/{name} returns one as a matrix you can post to /compute/bnw — use them to try a real network before inventing data.
  • /compute/fitqtl: how big an effect is, not just where. Give it the marker names a scan found (its peaks) and it fits R/qtl's fitqtl at each: columns marker, chr, pos, lod, pct_var, additive, se, t, p. Use it after /compute/rqtl whenever the user asks "how much does this QTL matter" — additive is half the difference between the two homozygote classes (the same convention GeneNetwork's published additive uses), pct_var the variance explained, and p the model's p-value. At most 100 markers per call; markers that cannot be fitted are absent, not zero. It fits covariates too — covariates: [{"case_attribute": "Sex"}], the same shape and the same coding as /compute/rqtl and /compute/gemma — so the additive it reports can be the effect after conditioning on a confounder, which is usually the number worth quoting. The reply names any covariate column it had to leave out because it does not vary among the phenotyped individuals (R/qtl would otherwise stop with "system is exactly singular"). Without covariates its lod at a marker equals /compute/rqtl's LOD at that marker; with one fitted it is a different model, so the two no longer match — quote the scan for "where", this for "how much".
  • /compute/qtl2 with "rqtl2_cross": "do": a multiparent scan — 8 founder alleles, an LMM with LOCO kinship. Only a cross that ships founder genotypes can do this (a .geno is a biallelic collapse); do is the one this deployment has. Pass samples by name; the LOD means "some founder differs", not "this strain differs" — per-founder effects are not returned yet. Minutes of genotype probabilities the first time, cached per cross.
  • /compute/qtl2: the same panel through R/qtl2 — genotype probabilities and an LMM with loco kinship ("kinship": "none" turns it into a Haley-Knott scan on probabilities). Rows are marker, chr, pos, lod, so /plot/rqtl draws them unchanged. Reach for it when a scan's peaks depend on relatedness, or when the panel is multiparent; it is slower than /compute/rqtl (a DO cross costs minutes of genotype probabilities, cached per panel). Autosomes only, and notes says what was left out.
  • Need a GEMMA threshold? This API refuses nperm on GEMMA (one LOCO scan per permutation, one GEMMA slot). Two routes: /compute/bulklmm with nperm (rotation null, seconds), or run GEMMA's own --permutate locally on the files the reply names (exact GEMMA null, hours). R/qtl's nperm is a sanity check only — its null ignores relatedness.
  • Need a threshold, not just a LOD? POST /compute/bulklmm (BulkLMM.jl) fits an LMM per marker with a global kinship matrix and permutes by rotation, so "nperm": 200 is cheap and its thresholds belong to the model that made the LODs (notes list alpha 0.10/0.05/0.01 and the seed). It is not GEMMA's LOCO model — say so when quoting a p-value — and h2_null near zero means it collapsed to a fixed-effects fit. One trait at a time is serial in Julia (the package threads only its many-trait scan); blocks of markers are scanned in parallel by the driver, so a reply's notes say how many ran.
  • Genotypes at a peak (the only genotypes the API returns): GET /groups/{g}/traits/{id}/peak-genotypes?marker=<marker> re-runs the scan, confirms the marker is at the peak (within ?window= Mb, LOD ≥ ?min_lod=) and returns one call per animal that has a value — 422 not_the_peak / not_a_hit otherwise. Plot it by posting that reply's genotypes with the trait's values to POST /plot/genotypes (R/qtl book Fig 4.1: genotype class across, value up).
  • .../markers/{marker}/classes?trait=<id> summarises a trait's values per genotype class (n, mean, sd, min, max) at one marker — how you see that a peak rests on two animals, or that a marker is nearly monomorphic after a conversion drops H. Genotypes never leave the API.
  • /genes/{symbol}/aliases returns Wikidata's alias list for a gene (cached seven days in gene-aliases.lmdb; ?species=human for the ortholog's labels). A "stale": true answer means Wikidata was unreachable and this is the last value -- not that the gene has no aliases.
  • /probesets/{name} finds every dataset carrying a probeset id (indexed; suffixed _A variants included, name says which matched) and /genes/{symbol}/traits lists a symbol's expression traits across datasets with a real total, ?sort=lod, ?min_lod=, ?group=. Neither returns values — a token is enough for the metadata, but values stay per-dataset: follow next to /datasets/{ds}/probesets/{name}/values.
  • /groups/{group}/genotypes/{version}/markers lists marker names only; .../markers/{marker}/location gives one marker's name/chr/cM/Mb. Use ?chr=&page=&limit=.
  • /groups/{group}/genotypes/{version}/frequencies (and .../markers/{marker}/frequencies for one locus; for a file no group lists, /groups/unlisted/genotypes/{name}/frequencies): how the panel is actually called — {"B": 0.49, "D": 0.49, "H": 0.02}. Use it before trusting a flat scan: a marker that is 99 % missing produces the same flat LOD as real data. Probability-valued files are binned 0.05 wide (?bin= to change); a file over 256 MB is refused (413) rather than read.
  • Compute replies carry the class-count hint at their own peak (.../markers/{peak}/classes?trait={trait_id}): filling in the trait id is how you check a peak before reporting it, and /compute/qtl2's notes explains what a riself conversion did to H.
  • Uploading new traits/covariates: POST /matrices with {"name","genotype_version","contents":"id,trait1,cov1\nBXD1,1.2,3\n…", "kinds":{"cov1":"covariate"}} — a delimited table whose first column names the individuals. It is checked against that genotype version's panel (same individuals, each named once) and refused with the difference otherwise. Each column becomes a basket <matrix>.<column>: map it with {"basket": "<matrix>.<column>"} on any compute endpoint. GET /matrices lists yours, GET /matrices/{name}/values reads the table back. scripts/example-matrix.py writes a valid example from a genotype file.
  • Uploading genotypes: POST /uploads/genotypes with {"name":"<user>-GENONAME","base_version":"BXD.4","contents":"<tsv>"} — tab-delimited, header = individuals from column 4, then marker/chr/pos. It updates a population: ≤50 new individuals, at least one overlapping, ≤200 000 markers, and the values must be the same kind as the base's (letters, whole numbers, probabilities). The reply's Every compute endpoint takes it by name — {"genotype_upload": "<name>"} on /compute/{rqtl,gemma,qtl2,fitqtl} — with samples (or a basket) matched by name; the server writes the form each engine reads. Uploading the same name again replaces it, and the digest in the reply says which version ran. A one-chromosome upload scans flat or empty under LOCO kinship — use "kinship": "overall" if that is all you have. scripts/example-genotype-upload.py makes a valid one from any .geno.
  • Changing values: there is no write endpoint for traits, and there never will be — but a user's own column can be stored and mapped. POST /baskets with {"name", "values":[{name,value}…]} (values may be numbers, strings like "NA", or null), PATCH /baskets/{name} to edit (a new revision, a new digest), then POST /compute/qtl2 with "basket": "<name>" instead of samples. Baskets are private to the token's user, live in their own SQLite file, and a mapping reply records the revision and digest it used — quote them when reporting a result. Each edit is a new revision with its own timestamp (GET /baskets/{name}/revisions), old revisions stay readable (?revision=N), and "basket": "name@2" maps a pinned one — quote name@revision when a result must be reproducible. All four compute endpoints take basket. Baskets live in one file per user under BASKET_DIR, named for the user, so a new token still finds them. GET /baskets lists them, DELETE removes one.
  • Follow next freely: every link is a GET that returns 200, and the API offers the whole graph from any response (genotypes -> frequencies -> markers -> a marker's location/tally -> PheWAS; trait -> values -> genotypes; /version -> every listing). actions holds the POSTs, which are not linkable.
  • /plot/rqtl, /plot/gemma: post the compute reply back, receive an image/svg+xml plot. Do not draw your own.
  • Options (all optional, add to the posted body; full list in components.schemas.RqtlRequest / GemmaRequest):
  • R/qtl: nperm (permutations; 100 takes about 14 s and adds significant/suggestive thresholds that /plot/rqtl then draws), method (hk default, ehk, mr, em, imp), model (normal default, np), control (a marker to condition on), genotype_version, include_parents_f1, covariates.
  • GEMMA: lmm (default 9), genotype_version, include_parents_f1, covariates, trait_name ("BXD::20010" — optional, but it is what the reply reports and what /plot/gemma titles the picture with; without it the title is <group>::gemma). nperm is refused (422 permutations_not_supported): for significance send the same values to /compute/rqtl with nperm.
  • Normalizing before mapping (both engines, same three fields): normalize = none | log2 | log10 | sqrt | negate | zscore | robust_zscore | rank (GN2's "Quantile"/qnorm; note it is a rank-based inverse-normal transform, not array quantile normalisation), plus winsor_quantile (e.g. 0.05 for the 5th/95th percentile) or winsor_fences (1.5 = GN2's outlier rule, Q1-1.5*IQR). They run transform → winsorise → standardise, and the reply's notes name the pipeline and any clipped/​outlying values. GET …/values?normalize= does the same for the values endpoint and carries the fields into the suggested mapping bodies, so use that when you want the fetched numbers to be the mapped ones. zscore is the population sd, matching genenetwork.org; effects (beta/additive) come out on the transformed scale, so say which scale you are quoting.
  • Covariates (/compute/rqtl, /compute/gemma and /compute/fitqtl): covariates: [{"case_attribute": "<name>"}]. Worth trying when the study's own metadata could confound the trait — sex, age at death, batch/epoch, cohort, sequencing depth — because they are fitted alongside the marker, and a peak that survives them is the stronger result. Read .data.descriptions before choosing one, not just .data.names: the names do not explain themselves, and the descriptions do (BXD: SeqCvge = "Depth of DNA sequencing for 1 male of each strain in 2016", Epoch = BXD family subgroups, Epoch1(BXD1-32), Epoch2-6 (BXD33-220), Status = Live= Available at JAX, Cryo=Cryopreserved only, Extinct, RRID = a SciCrunch identifier). BXD has no Sex; an unknown name is 422 covariate_unavailable listing the ones the group does have.
    • Then read the covariate line in notes. It reports the coding and the complete cases, e.g. Covariate Status (Live=…, Cryo=…): categorical coded Cryo=0, Cyro=1, Ext=2, Live=3, UTHSC=4, UTHSC2=5, 90 of 93 samples have a value, 34 with both the trait and the covariate; 6 levels. Report that n next to any LOD/p change — a peak that shifts because 60 samples left the fit is not a discovery. Note the data's own typo: Cyro and Cryo are two levels.
    • Categorical vs numeric. Numeric text passes through as numbers; anything else with 2–20 distinct values among the strains being mapped is coded 0..k−1 in sorted order; more than 20, a single level, or a map that misses a value is a 422 asking for an explicit map. Measured on BXD (trait 10001, BXD.4): RRID is refused — 422 … 82 distinct values, too many to code automatically — while Status (6 levels) and Epoch are accepted.
    • Compound values are a trap. SeqCvge mixes numbers with two-library values (40+31L) and carries 30 distinct values across the group: if few enough fall inside the mapped panel it is accepted, but as a categorical — and the GEMMA run then fails (502 gemma_failed) rather than telling you why. The same is true of any mostly-numeric attribute with a few odd entries. Send the coding yourself: {"case_attribute": "SeqCvge", "map": {"40+31L": 71, "40": 40, …}}, or inline {"name": …, "kind": "numerical", "values": […]}.
    • GEMMA gets k−1 dummy columns (first level = reference) and drops a level none of the fitted individuals carries; the reply says so (GEMMA fits 4 covariate column(s): Status.Ext, … and Covariate column(s) left out because they do not vary …). R/qtl takes the single level column and dummy-expands it itself, so its note has no such line.
    • Missing covariate values drop those individuals from the fit (both engines), which notes reports as Missing covariate values: ….
    • At most 8 covariates, and they need a group (the individuals are aligned by name, which a bare genotype_file request does not carry).
  • genotype_version: list them with /groups/{g}/genotypes (BXD.8 is current).
  • Timings seen: R/qtl about 7 s (1 MB reply), GEMMA about 3 s (1.9 MB), plots instant. They are synchronous; ?id= only adds progress logging.
  • Progress: GET /compute/{rqtl,gemma}/status/{id} returns lines and a pointer to pass back as peek; the last line is COMPLETION_MARKER or FAILURE_MARKER: <error>. It exists only for an id you passed on the POST (otherwise 404 run_not_found).
  • /plot/* accepts the whole compute reply or just its data; query params trait, title, and bonferroni=false (GEMMA only). The title names the trait from the reply's own Trait: ... note (both compute endpoints write it first), so posting the whole reply back is enough; ?trait= wins over the note.
  • Two traits really do draw two different plots (traces, peaks, annotation); the title is the one thing that was identical before the note existed.
  • /compute/correlation is a stub with canned data. Do not use it.
  • You never supply genotypes: the server uses its own files.

PheWAS (GET) /groups/{group}/phewas?locus=<marker> or ?chr=&mb=, window (bp, default 10000), genotype_version. Returns published traits whose recorded peak marker lies in the window, strongest first. An empty list is a real answer.

Data (GET, all need the token)

path gives you
/species, /species/{s}, /species/{s}/datasets species, expression datasets
/groups?species=, /groups/{g}, /groups/{g}/strains, /groups/{g}/datasets populations and samples
/groups/{g}/case-attributes the group's metadata: .data.names, and .data.descriptions saying what each one means — read those before fitting a covariate
/groups/{g}/traits, /groups/{g}/traits/{id}, /groups/{g}/traits/{id}/values published traits (the first is the bulk list)
/groups/{g}/traits/{id}/case-attributes/{attribute} a trait's case-attribute values, in the trait's own sample order, plus description and a distinct count — the honest way to see whether an attribute is usable as a covariate (a constant one is not)
/groups/{g}/genotypes[/{version}[/markers]] genotype versions and markers, and the marker names /compute/fitqtl takes
POST /compute/{rqtl,gemma,fitqtl} mapping (POST, not a GET): the scan, the LMM, and effect sizes at named markers — all three take covariates and normalize
/datasets/{ds}[/probesets[/{p}[/values]]], /traits/{ds}[/{t}[/values]] expression data
/publications/{pubmed_id}/traits traits a paper reports (PubMed id, not publication_id)
/accessions/{acc}, /mapping-methods GN accession lookup, methods offered

Bulk jobs. Fetch /groups/{g}/traits once (the whole PublishXRef list: id, phenotype_id, dataset, abbreviation, nsamples; about 15 000 for BXD), then .../traits/{id}/values per trait, a few at a time. Prefer one bulk call to a thousand small ones; do not use /search for bulk work.

Other examples: /datasets/Br_U_0803_M/probesets/100001_at (one expression trait), /publications/2760876/traits (PubMed id), and /search?q=cerebellum AND group:BXD&limit=5.

GeneNetwork2's trait page links out to a dozen services by gene symbol. These are the ones that still resolve (checked 2026-11-11, HTTP 200 and — where the page is not a JavaScript shell — the gene actually found), with the URL template to hand to a user. {SYMBOL} is the gene symbol; taxa are mouse 10090, rat 10116, human 9606.

service URL template what it is for
RGD (gene search) https://rgd.mcw.edu/rgdweb/elasticResults.html?term={SYMBOL}&category=Gene&species={Mouse\|Rat\|Human} the best single hub for mouse and rat: orthologs, phenotypes, expression, QTLs. Works for all three species
MGI https://www.informatics.jax.org/searchtool/SearchResults?query={SYMBOL} mouse gene authority: alleles, phenotypes, knockouts
Alliance of Genome Resources https://www.alliancegenome.org/search?category=gene&q={SYMBOL} cross-species reference: orthologs and disease models
NCBI Gene https://www.ncbi.nlm.nih.gov/gene/?term={SYMBOL}[gene]+AND+mouse[orgn] the canonical record; also the source of Gene IDs
Wikidata https://www.wikidata.org/wiki/{QID} identifiers and aliases across databases. The QID is what /genes/{SYMBOL}/aliases returns, so this link costs nothing to build
UniProt https://www.uniprot.org/uniprotkb?query=gene:{SYMBOL}+AND+organism_id:{TAXON} (or …/uniprotkb/{ACCESSION}) protein entry, domains, variants. Search by symbol, link by accession
Ensembl https://www.ensembl.org/Mus_musculus/Gene/Summary?g={ENSEMBL_ID} genome browser, transcripts, orthologs. Needs the Ensembl gene id
GTEx https://www.gtexportal.org/home/gene/{HUMAN_SYMBOL} human tissue expression. Human symbol only — a mouse symbol shows an empty report
STRING https://string-db.org/cgi/network?identifier={SYMBOL}&species={TAXON} protein–protein interaction network
GeneMANIA https://genemania.org/search/{mouse\|rat\|human}/{SYMBOL} functional association network, per species
Open Targets https://platform.opentargets.org/search?q={SYMBOL} target–disease associations (human)
Human Protein Atlas https://www.proteinatlas.org/search/{SYMBOL} human tissue/cell expression at protein level
EBI GWAS Catalog https://www.ebi.ac.uk/gwas/search?query={SYMBOL} published GWAS associations
BioGPS http://biogps.org/?org=10090#goto=genereport&id={SYMBOL} mouse expression across tissues and datasets
Allen Brain Atlas https://mouse.brain-map.org/search/show?search_type=gene&search_term={SYMBOL} in-situ hybridisation in mouse brain
GeneCup https://genecup.org/progress?type=GWAS&query={SYMBOL} literature-mined gene–disease links

Do not offer these (checked at the same time): HomoloGene (NCBI retired it in 2021 — the URL answers with a notice), UniGene (retired, HTTP 410), Genotation and GeneBridge (no DNS/connection), PhenoGen (404), the old chibi GEMMA gene page, wiki-Pi. PANTHER answers 403 to a bot but works in a browser, so offer it only as a "works in your browser" link: https://www.pantherdb.org/genes/geneList.do?searchType=basic&fieldName=all&organism=all&listType=1&fieldValue={SYMBOL}.

How to use them. Quote the symbol and the species you found it in, and let the user click. Do not synthesise what the other service says — say "RGD has a gene report for Shh (rat)" and give the link, not a paraphrase of a page you did not read. For a rat symbol prefer RGD and GeneMANIA(rat); for a mouse symbol all of the above except Human Protein Atlas and GWAS apply; for a human symbol use GTEx, Open Targets and the Protein Atlas.

The API builds them for you. GET /genes/{SYMBOL}/aliases returns a links array ([{name, href, description}]) built from the table above for the symbol and species it actually matched -- RGD, MGI, BioGPS and the Allen Brain Atlas for a mouse gene; RGD, GeneMANIA and UniProt for a rat one. It also returns label, the entity's canonical symbol in that species, which is how you get the human symbol a human-only service needs:

GET /genes/Shh/aliases?species=human
→ label "SHH", links […] including GTEx → https://www.gtexportal.org/home/gene/SHH

So a mouse Shh reaches GTEx through the human ortholog's label, and you never hold a URL template yourself. stale: true means Wikidata was unreachable and the cached entity was served -- the links still work, the label may be absent from an older cache entry (retry and it refreshes).

4. Gotchas that cost time

  • Compute replies are big (about 21 000 rows; GEMMA about 2 MB). Save to a file, inspect with jq / python3 -I. Piping to head forces a rerun.
  • Rows are positional; names are in meta.columns.
  • Not every value maps: trait 20010's 41 values map as 39, because C57BL/6J and DBA/2J are not in the BXD.8 genotype file, and include_parents_f1: true does not change that. /groups/{g}/strains flags such samples with used_for_mapping: false, which predicts it. notes on the reply says how many matched; tell the user.
  • A peak is often a plateau: neighbouring markers share the same LOD / p-value (trait 20010's GEMMA maximum, 5.79, is shared by several markers from 182.6 to 184.3 Mb). So the trait record's locus, R/qtl's peak, the plot summary and a reply's next can each name a different marker of the same strength. That is not an error: say "peak region" and use the marker the reply's next gives.
  • Before you believe a peak, count the animals in its genotype class. GET /groups/{g}/genotypes/{v}/markers/{marker}/classes?trait=<id> gives n, mean, sd, min, max per class at that marker (no genotypes, no per-animal values). A peak carried by 1–2 animals is a different claim from one carried by a hundred, and the reply says which classes are thin. On DOL trait 10003 GEMMA's top marker JAX00418132 (chr16 36.35 Mb, −log10 p 7.15) is A n=2 mean 1.8, H n=20 mean 2.53, B n=171 mean 3.95: the two A animals are the trait's two lowest values, and GEMMA's next two hits are the same pattern mirrored (1 A among 171, then 2 B among 171). That block is a haplotype carried by a handful of extreme animals, not a population effect — say so rather than reporting 7.15 as a locus.
  • When R/qtl finds a peak R/qtl2 doesn't, the model is the first suspect, not the data: R/qtl fits each marker's own calls (dropping the least frequent of three codes), R/qtl2 fits HMM probabilities with the H calls re-imputed. {"genotype_model": "f2"} keeps the het calls as a third genotype and shows what that recovers (BXD trait 16627: LOD 3.73 → 9.68 at rs33299937, against R/qtl's 9.59) — a diagnostic, not a result, since an inbred line has no heterozygotes. The reply says Cross: riself or Cross: f2: quote it.
  • H is dropped for R/qtl2 on a riself panel (@type:riset: BXD, DOL, most RI sets), and kept by GEMMA as dosage 0.5. R/qtl2 writes those cells as missing and its HMM re-imputes them from flanking markers, so the animals are not lost — the observation is. A marker that is mostly H becomes nearly monomorphic for R/qtl2 (2 A vs 171 B at the marker above) and cannot produce a peak there, which is why the two engines can name different markers of the same region. Check /frequencies or /classes before calling one engine wrong.
  • GEMMA's af is not an allele frequency. It is mean(dosage)/2 under the BIMBAM coding the deployment ships (B=1, H=0.5, A=0), so af 0.469 for JAX00418132 is half the B-allele frequency 0.935, not the A-allele frequency 0.065. Take allele frequencies from .../markers/{marker}/frequencies. Likewise allele1/allele0 in GEMMA's output are the BIMBAM X, Y placeholders, so the sign of beta is not a strain.
  • Do not compare scores across engines as if they were the same scale. GEMMA reports −log10 p, R/qtl and R/qtl2 report LOD, qtlreaper reports LRS (lod = lrs / 4.604). The same region at "7.15" and "3.92" is not a disagreement, and an LMM's LOD is not a Haley-Knott LOD.
  • PheWAS window matters: at the trait-20010 peak, 10 kb finds 4 traits and 1 Mb finds 19. The reply's next offers a ten-times wider window.
  • Public-data rule: public > 0 and confidentiality < 1, or older than five years; otherwise 403, do not retry. Metadata is visible even when values are not.
  • There are no write endpoints, and the search index holds public documents only.
  • Older docs said POST /compute/* is refused with 403. On this build it works. If you do get a 403 there, that is the answer: do not work around it. If the task needs private data or computation you are not granted, stop and ask a human: that needs an account and a grant, not a different URL.

/search?q= is Xapian: juxtaposed terms are OR, so write AND; free text is stemmed and case-insensitive; quoted phrases, wildcards and hyphenated words match nothing. Fields: exact group, dataset, species, author, tissue, symbol, chr, peakchr, position; stemmed description, rif, wiki; ranges mean, peak, peakmb, additive, year (field:lo..hi). Aliases: pos for position, RIF for rif, WIKI for wiki; Hs/Mm are species-location shorthands. Values are lowercased, so case does not matter. A prefix whose value is not in the index matches nothing; that is "no such value", not an error. NEAR/ADJ do not work either. A dangling operator or unbalanced quote is 422 invalid_query with Xapian's reason in detail: fix it, do not retry. type=phenotype|gene, limit up to 100, page. There is no total: counting means paging, and each page costs time proportional to the matches left, so do not page through thousands of hits. For bulk work use /groups/{g}/traits.

.data is an array of hits. Each has name (the trait id, e.g. "21614"), dataset, dataset_fullname, group, species, authors, year, description, mean and more. Without group: you also get other groups' phenotypes (for example C57BL-6JxC57BL-6NJF2Publish), whose ids only mean something inside their own group; add AND group:BXD to stay in BXD.

6. Tokens and errors

A GeneNetwork team member issues the token by hand; it cannot be self-served. If you have none, ask: "To read GeneNetwork data I need a GeneNetwork API token (a JWT) for https://vesuvio.pu.ac.ke/gn4. It is issued by hand; it helps to say who it is for, which groups/datasets, how long (default 30 days) and what for." Ask the user to pass it out of band, not in a public issue, a commit or a shared chat. Send it as Authorization: Bearer <token> on every request except /version and /openapi.json, and keep it for the whole session. Never put it in a URL, a commit, a log line or anything shown to someone else, and do not hand it to other users or paste it into shared state.

status do
401 invalid_token detail says ExpiredSignature (expired) or InvalidIssuer (wrong deployment). You cannot renew it: tell the user, ask for a fresh one quoting the request_id, retry once, then stop and report; do not loop or look for another credential
403 forbidden not public: do not retry
404 not_found check the name; next lists valid values
422 invalid_query, invalid_lmm fix the query or parameter; do not retry
429 back off and honour Retry-After (not emitted today, but in the contract)
502/503 transient: back off, max about 5 tries

Errors are RFC 7807-style JSON (type, title, status, code, detail, operation_id, spec_url, request_id, sometimes next); type, spec_url and next are absolute paths under /gn4/. Quote request_id when reporting a problem (also in the x-request-id header). No rate limit is configured, but be considerate: a few concurrent requests, no tight polling loops.

7. Keep this skill current: reload it if GeneNetwork changes mid-session

Do not rely on a version written into this file. Instead, remember the build you saw at the start of the session and compare against it later.

  1. At the start (section 0), note curl -s https://vesuvio.pu.ac.ke/gn4/version | jq -r .data.git as your session baseline.
  2. Check /version again when something behaves differently from what is written here (an unexpected 404, a new field in actions, a path in openapi.json this file does not mention), and before a long or repeated piece of work.
  3. If the revision is no longer the baseline, GeneNetwork was updated while you worked. Suggest to the user that you reload SKILL.md from the repo (docs/www/SKILL.md in https://git.genenetwork.org/genenetwork4; a sparse clone of docs is enough), and use that copy from then on. One line is enough, for example: "GeneNetwork was updated since we started ( -> ); I'd reload SKILL.md from docs/www/ to pick up any changes. Shall I?" Then set the new revision as your baseline.
  4. If the repo is unreachable from the sandbox, fall back to /gn4/openapi.json, which always matches the running server, and tell the user the skill file itself could not be refreshed.

7a. Project memory: the answers this repo has already worked out

This repository carries an ai-memory store (ai-memory.sqlite) linking each commit to the reasoning behind it, plus an optional Xapian topic index over both. A previous session's context window is gone; this is what survives it, and it is cheaper than re-deriving anything:

ai-memory -C genenetwork4 search 'pve'          # SQLite FTS over the reasoning: ~15 ms
ai-memory -C genenetwork4 why gemma.compute_gemma   # commits touching a symbol, with its reasoning
ai-memory -C genenetwork4 --index DIR search 'topic:pve repo:genenetwork4'   # topic index: ~100 ms

topic: terms come from the session id, so topic:ravanan-slurm, topic:reference-genomes, topic:gemma-pve are the entry points that exist.

Two things to look for before starting work that touches an engine, a cache or a cluster:

  • a RUNBOOK entry — the operational ones carry the actual commands, ports and fingerprints, e.g. bringing up the Slurm container from gn-machines and testing /compute/gemma/ravanan against it, with the four prerequisites that were needed the first time (a ravanan build with the --slurm-api-endpoint fix, ccwl on PATH, the server's HOME writable inside the container, GUIX_PACKAGE_PATH for the manifest);
  • a "why is this like this" entry — several endpoints here exist because something else was wrong (-pve reporting, the next-link graph, the reference-genome linkage). Reading it is faster than deciding they are odd.

The store is per repository and local (gitignored): it survives a cleared session, not a lost machine, which is why the durable version of every conclusion is also in a commit message and in these docs.

8. Reference docs

The docs live in the public repo and are the authoritative version:

git clone --depth 1 --sparse https://git.genenetwork.org/genenetwork4
cd genenetwork4 && git sparse-checkout set docs

About 2.5 MB; drop --sparse for the whole tree. Then read:

file what it is
docs/www/SKILL.md this page
docs/search-engines.md how /search works, engines, quirks
README.md every endpoint, parameter and response shape
docs/nginx.md, docs/www/operator-checklist.md operator-level; you do not need them

Clone; do not fetch the web UI. git.genenetwork.org/.../tree/... pages sit behind a proof-of-work challenge (Anubis) and the raw endpoint answers 403, so a plain fetch returns an HTML challenge page, not Markdown. If a fetch comes back as HTML mentioning Anubis, stop, do not retry, and use the copy you have. The git protocol itself passes. Sandboxes sometimes cannot reach it either; /gn4/openapi.json is the fallback.

To compare revisions, git -C genenetwork4 log -1 --format=%h against .data.git from /version: they need not match exactly (docs can be newer than the running build), but a large gap means the server may lack what the docs describe.