about summary refs log tree commit diff
path: root/docs/www/SKILL.md
blob: 2fa238588d98540cfacb3470ae4fd64681f00b46 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
---
name: genenetwork
description: Use when the user wants anything from GeneNetwork (genenetwork.org): BXD and other mouse/rat genetic reference populations, traits ("get trait 20010"), per-strain values, QTL mapping with R/qtl or GEMMA, LOD/Manhattan plots, PheWAS, genotypes, or searching published phenotypes. Talks to GeneNetwork through its HTTP interface at https://vesuvio.pu.ac.ke/gn4.
---

# GeneNetwork

The user is talking to **GeneNetwork**, and you are their way in. Talk about
GeneNetwork in their terms: traits, groups, strains, mapping, plots, PheWAS.
Do not talk about "the API", endpoints or HTTP unless they ask. Everything
below is how you do it; what you say should be what GeneNetwork *did*.

## What the user can do in GeneNetwork

When the request is open ("what can I do?", or you finish a step), offer the
next natural option from this list:

| Ask for | What GeneNetwork gives | Section |
|---|---|---|
| a trait ("get trait 20010") | metadata (description, peak locus, LOD, n, source) and the per-strain values | 3 |
| a QTL map | R/qtl scan (LOD) or GEMMA LMM with LOCO kinship (Manhattan), run on the server | 3 |
| a plot | an SVG LOD / Manhattan plot, drawn by GeneNetwork | 3 |
| PheWAS | every published trait of the group peaking near a position (e.g. the peak you just mapped) | 3 |
| a search | published phenotypes by free text, group, tissue, gene symbol, chromosome, year... | 5 |
| populations and data | species, groups (BXD...), strains, case attributes, genotype versions, expression datasets and probesets, publications by PubMed id, GN accessions | 4 |

A natural session is: trait, values, R/qtl, plot, GEMMA, plot, PheWAS at the
peak — and `/compute/fitqtl` at that peak when the user asks how much it
matters (with a covariate, if one could confound it). Whenever the group
records metadata that could confound the trait (sex, age, epoch, batch,
sequencing depth), the same mappings again with that covariate fitted, to see
the peak survive it. After each step, suggest the next one.

## 0. Do this first

1. **Check you can reach GeneNetwork** (no token needed):
   `curl -s https://vesuvio.pu.ac.ke/gn4/version`
   and, for the docs repo, `git ls-remote https://git.genenetwork.org/genenetwork4 HEAD`.
   A sandbox proxy blocks any host that is not on the person's allowlist
   (`connect_rejected`, or `CONNECT tunnel failed, 403` from git). **Ask them to
   allow both domains in one go**, not one at a time:

   > To work with GeneNetwork I need two domains allowed for this session:
   > `vesuvio.pu.ac.ke` (the data and mapping service) and `git.genenetwork.org`
   > (the docs, so I can reload this skill when it changes). Please add them under
   > claude.ai > Settings > Capabilities (admins: Organization settings >
   > Capabilities), then start a new message or session.

   If only the repo is blocked, GeneNetwork itself still works: carry on, and
   tell the user the skill file cannot be refreshed (section 7). Say all this once
   and stop probing other routes.
2. **Token.** Look for one already given to you in the task context. If there
   is none, stop and ask (section 6). Never guess, invent, reuse a token that
   was not given to you, or look for another way in: `POST /gn4/get-test-token`
   is a deliberate 404, and there is no sign-up or login. A token carries no
   rights of its own; what you may read is decided per dataset.
3. **Note the build** (section 7): remember `.data.git` from `/version` as
   your baseline; if it changes later in the session, suggest reloading
   `docs/www/SKILL.md`.
4. **Read the spec, not only this page.** `GET /gn4/openapi.json` lists every
   operation and is what the server really runs; this page can lag. Use
   `jq -r '.paths | keys[]'` before saying something is not available. The
descriptions inside it are the real reference: every request option is
documented in `components.schemas` (for example
`jq '.components.schemas.RqtlRequest.properties' openapi.json`).
5. **Assume, don't ask.** A bare numeric trait id (`20010`) is a published
   phenotype of group **BXD** (dataset `BXDPublish`). Try it; ask only on 404.

Base URL for everything: `https://vesuvio.pu.ac.ke/gn4`. `/gn4` without the
trailing slash redirects there; anything outside `/gn4/` is a 404 from the
front end, not from GeneNetwork. Only `/version` and `/openapi.json` need no
token; everything else answers 401 (no token) or 404 (not routed).

## 1. Read the replies: GeneNetwork tells you what to do next

Every JSON reply is `{"data","meta","notes","next"}` and sometimes `"actions"`.

* `notes`: read them. They say how many strains were matched, what was run,
  and caveats.
* `next`: GET links to related things. Each `href` already includes `/gn4`:
  resolve against the origin (`https://vesuvio.pu.ac.ke$href`).
* **`actions`: ready-to-send POSTs.** On a trait's `/values` they are the R/qtl
  and GEMMA calls with the body already filled in; on a **compute reply** they
  are the matching plot call (href with `?trait=` already set; post the reply
  back to it). Use them instead of building calls by hand. Select by href, not
  position: `jq '.actions[] | select(.href|endswith("/compute/rqtl")) | .body'`.
* **Every compute reply also has `next`**: a PheWAS link at that run's peak
  marker. Read `next` and `actions` of each reply before deciding what to do.
* `meta`: `state` (`complete`/`partial`), `took_ms`, `request_id`, plus
  `columns`/`num_rows` on tabular payloads.
* `meta.state: partial` means usable but incomplete; the reason is in `notes`.
  Do not read a `null` enrichment field as "no such value in GeneNetwork"; read
  it as "ask again later".
* Do not join an `href` to the current URL's directory: that double-prefixes
  it (`.../gn4/gn4/...`).

### Always offer the next step

**End every answer with one sentence saying what to do next**, built from that
reply's own `next`/`actions` — never from an idea of what a mapping pipeline
"usually" does. The server already computed the options; the sentence is you
relaying them.

* **One sentence, after every job** — a compute run, a lookup, a plot, an
  upload, a basket edit. Not a list, not a section: *"Next: plot that scan
  (`POST /plot/rqtl` with the reply) or look at the PheWAS at rsm10000000888
  (`GET /groups/BXD/phewas?locus=rsm10000000888`)."*
* **Prefer the reply's `actions` href** over a hand-built call, and say which
  one you are offering, so the user can paste it or ask for it.
* **A refusal is an answer too.** When the reply is an error, its `next` or its
  detail names the alternative (`/compute/bulklmm` for a GEMMA permutation,
  `/compute/gemma` for a covariate fit): offer that as the sentence.
* **Do not stack suggestions.** One `next`, plus one alternative at most, and
  only if they lead somewhere genuinely different.
* **The user can switch it off.** "Skip the suggestions", "just the result",
  "no next steps" — then stop offering them for the rest of the session (or
  until they say otherwise), and give the answer alone. Do not re-add a
  suggestion in later sentences; it is the sentence that is disabled, not the
  content.
* **Never invent a step the API cannot do.** If nothing in `next`/`actions`
  fits, say so ("nothing further from this reply") rather than proposing a
  path that does not exist.

## 2. Workflow: map a trait and plot it

```sh
API=https://vesuvio.pu.ac.ke/gn4; AUTH="Authorization: Bearer $TOKEN"; D=$(mktemp -d)
curl -s -H "$AUTH" $API/groups/BXD/traits/20010                       # metadata
curl -s -H "$AUTH" $API/groups/BXD/traits/20010/values > $D/v.json    # values + actions
jq '.actions[]|select(.href|endswith("/compute/rqtl"))|.body' $D/v.json > $D/body.json

# R/qtl. Save the FULL reply to a file (never pipe through head), then plot it.
curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/body.json \
  $API/compute/rqtl > $D/rqtl.json
curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/rqtl.json \
  -D $D/h.txt -o rqtl.svg "$API/plot/rqtl?trait=BXD::20010"; grep -i x-plot-summary $D/h.txt

# GEMMA: same shape. Use /compute/gemma/ravanan -- it is the default runner
# (see "Which GEMMA endpoint" below).
jq '.actions[]|select(.href|endswith("/compute/gemma"))|.body' $D/v.json > $D/gbody.json
curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/gbody.json \
  "$API/compute/gemma/ravanan?id=t20010" > $D/gemma.json
curl -s -H "$AUTH" -H 'Content-Type: application/json' --data @$D/gemma.json \
  -o gemma.svg "$API/plot/gemma?trait=BXD::20010"

# PheWAS at the peak: follow the compute reply's own `next` (10 kb window), or widen it
jq -r '.next[0].href' $D/rqtl.json
curl -s -H "$AUTH" --get --data-urlencode locus=<peak marker> --data window=1000000 \
  $API/groups/BXD/phewas              # window = half-width in bp
```

Give the user the SVGs (they render directly) and quote the `x-plot-summary`
header (peak, and the reference lines drawn when you asked for them).

### Short API tokens (`gn4ai_...`)

A second credential format beside gn-auth's JWT, for agents and scripts that
should not carry a user's password or a long-lived JWT. Opaque: nothing is
stored inside it, and everything about it lives in the server's registry (one
SQLite database, `tokens.sqlite`).

- **Administer them from this machine only.** `scripts/gn4-token.py
  create|list|revoke|who` (routes `/tokens`, `/tokens/{id}`, `/tokens/who`).
  Those routes answer only a direct caller on this machine — the server checks
  the TCP peer address, not a header — and a request that arrives through a
  proxy is refused, so this can never be driven from the network.
- **The token is shown exactly once**, by `create`. Only its SHA-256 is stored;
  there is no way to recover or enumerate one. `list` shows id, subject, scope,
  status, prefix, expiry and last use.
- **Send it as `Authorization: Bearer gn4ai_...`** — never as a query
  parameter, which would land in nginx's log and in `next` links.
- **Public data only.** It works on every token-only endpoint (search, compute,
  plots, genotype metadata and frequencies, baskets, uploads, matrices) and on
  secured endpoints for public rows; a confidential dataset answers 403 telling
  the caller to use a gn-auth JWT, because gn-auth cannot validate an opaque
  token.
- **Revocation takes effect on the next request**, with no restart: it is a row
  in the registry. Expiry is fixed when minted (30 days by default).
- 64 characters, `gn4ai_` + 56 body characters + a 2-character checksum, from an
  alphabet with no look-alikes (`0/O`, `1/l/I`) — a typo is refused before the
  registry is asked.
- **In the log**, one JSON line per request names the person and the credential:
  `{"event":"request","ts":…,"user":"zachs","scheme":"opaque","token":"agent-1"}`
  — never the token or its hash. `scripts/gn4-log.py -f LOG` renders that as a
  table with the handle resolved against the registry, so it shows the subject
  as the registry has it now and the token's status *today* (`[revoked]`,
  `[expired]`, `[unknown]`); `--json` enriches the records for `jq`, and
  `--summary` counts requests per caller and credential.

### Which GEMMA endpoint: ravanan by default

Two endpoints run the same computation, and the reply shape is identical:

| endpoint | runs |
|---|---|
| `/compute/gemma/ravanan` | `gemma` itself, one invocation per chromosome, scattered by ravanan (Slurm) |
| `/compute/gemma` | `gemma-wrapper` (kinship, then GWA, both `--loco`) as a server subprocess, no cluster |

**Use `/compute/gemma/ravanan` by default.** Reach for `/compute/gemma` only
when:

- **Slurm is down or job submission fails** (`502`/`ravanan_failed`, or a job
  that never leaves `PENDING`) -- the in-process runner needs no cluster;
- **the user asks for it**, or the request needs something only it supports
  (covariates, normalization -- ravanan refuses those with a `422`);
- **you are testing or verifying results.** The wrapper endpoint is the
  reference: send the same body to both and compare rows. `scripts/gemma-compare.py
  --base URL --body FILE` does exactly that, prints both wall-clock times, and
  exits non-zero if any `rs` disagrees on `beta`, `se`, `l_mle` or `p_lrt`.

Both are warm-cached and both are cheap to repeat: a repeat of an identical
request comes back from cache (ravanan's store, or gemma-wrapper's
`--cache-dir`) in ~1-2 s against ~12-15 s for a first run, so re-running to
verify costs almost nothing.

## 3. GeneNetwork functionality, in detail

**Mapping and plots (POST, verified working)**

* `/compute/rqtl`: R/qtl `scanone` (Haley-Knott) over the group's genotypes.
  Columns `name, chr, pos, lod`. `nperm=0` by default, so no permutation lines.
* `/compute/gemma`: GEMMA LMM with leave-one-chromosome-out kinship. Columns
  `chr, rs, ps, n_miss, allele1, allele0, af, beta, se, l_mle, additive, p_lrt`.
  Pass `?id=` and poll `/compute/gemma/status/{id}` for long runs. Per-marker
  p-values only; the plot draws no reference line unless asked —
  `?bonferroni=true` for GEMMA's own, `?thresholds=` for another engine's.
  **Effect size is in the row, next to the score**: `beta` (GEMMA's effect per
  maternal-allele copy) with `se`, and `additive` = `−beta/2`, which is the
  number GeneNetwork's own tables and CSV show. Quote `additive` with its sign
  ("each maternal-allele copy lowers the trait by X" / the other way) and
  remember it is *this run* on these samples, not the `additive` in trait info,
  which came from the published mapping.
  A note also carries GEMMA's own variance component for the null model
  (`-pve`): under LOCO there is one per chromosome, and the note lists **every**
  one — `chr1 0.6674±0.3451, chr2 0.8426±0.3668, …` — with the range and the
  two extremes (plus their `vg`/`ve`) spelled out. Quote per-chromosome values,
  not an average: they are twenty different kinship matrices.
  `pve` is the share of variance the kinship explains — the same quantity as
  `/compute/bulklmm`'s `h2`, per chromosome instead of genome-wide — so it is
  the honest way to check the two engines against each other. It is
  scale-free: compare `pve` with `h2`, never `vg` with `σ²_e`. A `pve` of
  exactly 0 or 1 is a boundary (the standard error then reads "not
  available"), a direction rather than a proportion.
* **Choosing individuals**: every group-mapping endpoint (`/compute/gemma`,
  `/compute/gemma/ravanan`, `/compute/rqtl`, `/compute/qtl2`, `/compute/fitqtl`,
  `/compute/bulklmm`) excludes parents and F1s by default. `include_parents_f1:
  true` keeps them all; `individuals: {include: [names], exclude: [names]}`
  selects by name, `exclude` winning, with an unknown name a 422 that lists the
  panel. Remember *including them changes the kinship, not the fit* — GEMMA and
  R/qtl compute kinship without phenotypes, so parents/F1s shape relatedness
  while the association uses only animals with values; the reply's Genotype note
  gives both counts.
* `/compute/bnw`: a Bayesian network over a set of traits (BNW). Give `group` +
  `traits`, or `values` (one entry per variable with per-sample values). Columns
  `from, to, posterior, in_structure`; one row per candidate edge, strongest
  first. **A posterior is not a p-value** — it is the share of the k-best
  networks containing the edge, i.e. how often the search drew that arrow in
  this sample, so quote it as a structural claim and never as a threshold on a
  QTL. Parameters default to BNW's own (maxparents 4, k 1, tiers 3, threshold
  0.5) and the reply names the ones used. At most 40 variables, at least 3
  samples complete across every variable. `POST /plot/bnw` draws it from the
  posted reply — the network by default, `?kind=posterior` for the matrix of
  every candidate edge (arrow width and label are the posterior; dotted lines
  are edges that did not clear the threshold).
  `POST /compute/bnw/features` turns the network into traits: each node with
  parents becomes a module (node + parents) scored by the first principal
  component of its standardised members; `?module=NAME` returns that module's
  `{name, value}` scores, which go straight into `/compute/rqtl` or
  `/compute/gemma` as `samples`. `GET /compute/bnw/examples` lists BNW's own
  worked examples, and `/compute/bnw/examples/{name}` returns one as a matrix
  you can post to `/compute/bnw` — use them to try a real network before
  inventing data.
* `/compute/fitqtl`: **how big** an effect is, not just where. Give it the
  marker names a scan found (its peaks) and it fits R/qtl's `fitqtl` at
  each: columns `marker, chr, pos, lod, pct_var, additive, se, t, p`. Use it
  after `/compute/rqtl` whenever the user asks "how much does this QTL
  matter" — `additive` is half the difference between the two homozygote
  classes (the same convention GeneNetwork's published additive uses),
  `pct_var` the variance explained, and `p` the model's p-value. At most 100
  markers per call; markers that cannot be fitted are absent, not zero.
  **It fits covariates too** — `covariates: [{"case_attribute": "Sex"}]`,
  the same shape and the same coding as `/compute/rqtl` and
  `/compute/gemma` — so the `additive` it reports can be the effect *after*
  conditioning on a confounder, which is usually the number worth quoting.
  The reply names any covariate column it had to leave out because it does
  not vary among the phenotyped individuals (R/qtl would otherwise stop
  with "system is exactly singular"). Without covariates its `lod` at a
  marker equals `/compute/rqtl`'s LOD at that marker; with one fitted it is
  a different model, so the two no longer match — quote the scan for
  "where", this for "how much".
* `/compute/qtl2` with `"rqtl2_cross": "do"`: a **multiparent** scan — 8
  founder alleles, an LMM with LOCO kinship. Only a cross that ships founder
  genotypes can do this (a `.geno` is a biallelic collapse); `do` is the one
  this deployment has. Pass `samples` by name; the LOD means "some founder
  differs", not "this strain differs" — per-founder effects are not returned
  yet. Minutes of genotype probabilities the first time, cached per cross.
* `/compute/qtl2`: the same panel through **R/qtl2** — genotype
  probabilities and an LMM with `loco` kinship (`"kinship": "none"` turns it
  into a Haley-Knott scan on probabilities). Rows are `marker, chr, pos, lod`,
  so `/plot/rqtl` draws them unchanged. Reach for it when a scan's peaks
  depend on relatedness, or when the panel is multiparent; it is slower than
  `/compute/rqtl` (a DO cross costs minutes of genotype probabilities, cached
  per panel). Autosomes only, and `notes` says what was left out.
* **Need a GEMMA threshold?** This API refuses `nperm` on GEMMA (one LOCO scan
  per permutation, one GEMMA slot). Two routes: `/compute/bulklmm` with
  `nperm` (rotation null, seconds), or run GEMMA's own `--permutate` locally on
  the files the reply names (exact GEMMA null, hours). R/qtl's `nperm` is a
  sanity check only — its null ignores relatedness.
* **Need a threshold, not just a LOD?** `POST /compute/bulklmm` (BulkLMM.jl)
  fits an LMM per marker with a global kinship matrix and permutes by rotation,
  so `"nperm": 200` is cheap and its thresholds belong to the model that made the
  LODs (`notes` list alpha 0.10/0.05/0.01 and the seed). It is **not** GEMMA's
  LOCO model — say so when quoting a p-value — and `h2_null` near zero means it
  collapsed to a fixed-effects fit. One trait at a time is **serial** in
  Julia (the package threads only its many-trait scan); blocks of markers are
  scanned in parallel by the driver, so a reply's `notes` say how many ran.
* **Genotypes at a peak** (the only genotypes the API returns):
  `GET /groups/{g}/traits/{id}/peak-genotypes?marker=<marker>` re-runs the
  scan, confirms the marker is at the peak (within `?window=` Mb, LOD ≥
  `?min_lod=`) and returns one call per animal **that has a value** —
  `422 not_the_peak` / `not_a_hit` otherwise. Plot it by posting that reply's
  `genotypes` with the trait's values to `POST /plot/genotypes` (R/qtl book
  Fig 4.1: genotype class across, value up).
* `.../markers/{marker}/classes?trait=<id>` summarises a trait's values per
  genotype class (`n`, `mean`, `sd`, `min`, `max`) at one marker — how you
  see that a peak rests on two animals, or that a marker is nearly
  monomorphic after a conversion drops `H`. Genotypes never leave the API.
* `/genes/{symbol}/aliases` returns Wikidata's alias list for a gene
  (cached seven days in `gene-aliases.lmdb`; `?species=human` for the
  ortholog's labels). A `"stale": true` answer means Wikidata was
  unreachable and this is the last value -- not that the gene has no
  aliases.
* `/probesets/{name}` finds every dataset carrying a probeset id (indexed;
  suffixed `_A` variants included, `name` says which matched) and
  `/genes/{symbol}/traits` lists a symbol's expression traits across
  datasets with a real total, `?sort=lod`, `?min_lod=`, `?group=`. Neither
  returns values — a token is enough for the metadata, but values stay
  per-dataset: follow `next` to `/datasets/{ds}/probesets/{name}/values`.
* `/groups/{group}/genotypes/{version}/markers` lists marker **names**
  only; `.../markers/{marker}/location` gives one marker's
  `name`/`chr`/`cM`/`Mb`. Use `?chr=&page=&limit=`.
* `/groups/{group}/genotypes/{version}/frequencies` (and
  `.../markers/{marker}/frequencies` for one locus; for a file no group
  lists, `/groups/unlisted/genotypes/{name}/frequencies`): how the panel is
  actually called — `{"B": 0.49, "D": 0.49, "H": 0.02}`. Use it before
  trusting a flat scan: a marker that is 99 % missing produces the same
  flat LOD as real data. Probability-valued files are binned 0.05 wide
  (`?bin=` to change); a file over 256 MB is refused (413) rather than
  read.
* Compute replies carry the class-count hint at their own peak
  (`.../markers/{peak}/classes?trait={trait_id}`): filling in the trait id is
  how you check a peak before reporting it, and `/compute/qtl2`'s `notes`
  explains what a `riself` conversion did to `H`.
* **Uploading new traits/covariates**: `POST /matrices` with
  `{"name","genotype_version","contents":"id,trait1,cov1\nBXD1,1.2,3\n…",
  "kinds":{"cov1":"covariate"}}` — a delimited table whose first column names
  the individuals. It is checked against that genotype version's panel (same
  individuals, each named once) and refused with the difference otherwise.
  Each column becomes a basket `<matrix>.<column>`: map it with
  `{"basket": "<matrix>.<column>"}` on any compute endpoint. `GET /matrices`
  lists yours, `GET /matrices/{name}/values` reads the table back.
  `scripts/example-matrix.py` writes a valid example from a genotype file.
* **Uploading genotypes**: `POST /uploads/genotypes` with
  `{"name":"<user>-GENONAME","base_version":"BXD.4","contents":"<tsv>"}` —
  tab-delimited, header = individuals from column 4, then marker/chr/pos.
  It **updates** a population: ≤50 new individuals, at least one overlapping,
  ≤200 000 markers, and the values must be the same kind as the base's
  (letters, whole numbers, probabilities). The reply's
  Every compute endpoint takes it by name — `{"genotype_upload": "<name>"}` on
  `/compute/{rqtl,gemma,qtl2,fitqtl}` — with `samples` (or a `basket`) matched
  **by name**; the server writes the form each engine reads. Uploading the same
  name again **replaces** it, and the digest in the reply says which version
  ran. A one-chromosome upload scans flat or empty under LOCO
  kinship — use `"kinship": "overall"` if that is all you have. `scripts/example-genotype-upload.py`
  makes a valid one from any `.geno`.
* **Changing values**: there is no write endpoint for traits, and there never
  will be — but a user's own column can be stored and mapped. `POST /baskets`
  with `{"name", "values":[{name,value}…]}` (values may be numbers, strings
  like `"NA"`, or null), `PATCH /baskets/{name}` to edit (a new revision, a new
  digest), then `POST /compute/qtl2` with `"basket": "<name>"` instead of
  `samples`. Baskets are private to the token's user, live in their own SQLite
  file, and a mapping reply records the revision and digest it used — quote
  them when reporting a result. Each edit is a new revision with its own
  timestamp (`GET /baskets/{name}/revisions`), old revisions stay readable
  (`?revision=N`), and `"basket": "name@2"` maps a pinned one — quote
  `name@revision` when a result must be reproducible. All four compute
  endpoints take `basket`. Baskets live in one file per user under
  `BASKET_DIR`, named for the user, so a new token still finds them.
  `GET /baskets` lists them, `DELETE` removes one.
* Follow `next` freely: every link is a `GET` that returns 200, and the API
  offers the whole graph from any response (genotypes -> frequencies ->
  markers -> a marker's location/tally -> PheWAS; trait -> values ->
  genotypes; `/version` -> every listing). `actions` holds the `POST`s, which
  are not linkable.
* `/plot/rqtl`, `/plot/gemma`: post the compute reply back, receive an
  `image/svg+xml` plot. Do not draw your own.
* Options (all optional, add to the posted body; full list in
  `components.schemas.RqtlRequest` / `GemmaRequest`):
  * R/qtl: `nperm` (permutations; **100 takes about 14 s** and adds
    `significant`/`suggestive` thresholds that `/plot/rqtl` then draws),
    `method` (`hk` default, `ehk`, `mr`, `em`, `imp`), `model` (`normal`
    default, `np`), `control` (a marker to condition on), `genotype_version`,
    `include_parents_f1`, `covariates`.
  * GEMMA: `lmm` (default 9), `genotype_version`, `include_parents_f1`,
    `covariates`, `trait_name` (`"BXD::20010"` — optional, but it is what
    the reply reports and what `/plot/gemma` titles the picture with;
    without it the title is `<group>::gemma`). `nperm` is refused (`422 permutations_not_supported`): for
    significance send the same values to `/compute/rqtl` with `nperm`.
  * **Normalizing before mapping** (both engines, same three fields):
    `normalize` = `none` | `log2` | `log10` | `sqrt` | `negate` | `zscore` |
    `robust_zscore` | `rank` (GN2's "Quantile"/`qnorm`; note it is a
    rank-based inverse-normal transform, not array quantile normalisation),
    plus `winsor_quantile` (e.g. `0.05` for the 5th/95th percentile) or
    `winsor_fences` (`1.5` = GN2's outlier rule, `Q1-1.5*IQR`). They run
    transform → winsorise → standardise, and the reply's `notes` name the
    pipeline and any clipped/​outlying values. `GET …/values?normalize=` does
    the same for the values endpoint and carries the fields into the
    suggested mapping bodies, so use that when you want the fetched numbers
    to be the mapped ones. `zscore` is the population sd, matching
    genenetwork.org; effects (`beta`/`additive`) come out on the transformed
    scale, so say which scale you are quoting.
  * **Covariates** (`/compute/rqtl`, `/compute/gemma` **and `/compute/fitqtl`**):
    `covariates: [{"case_attribute": "<name>"}]`.
    Worth trying when the study's own metadata could confound the trait —
    sex, age at death, batch/epoch, cohort, sequencing depth — because they
    are fitted alongside the marker, and a peak that survives them is the
    stronger result. **Read `.data.descriptions` before choosing one**, not
    just `.data.names`: the names do not explain themselves, and the
    descriptions do (BXD: `SeqCvge` = "Depth of DNA sequencing for 1 male of
    each strain in 2016", `Epoch` = BXD family subgroups, `Epoch1(BXD1-32)`,
    `Epoch2-6 (BXD33-220)`, `Status` = `Live= Available at JAX,
    Cryo=Cryopreserved only, Extinct`, `RRID` = a SciCrunch identifier).
    BXD has no `Sex`; an unknown name is `422 covariate_unavailable` listing
    the ones the group does have.
    * **Then read the covariate line in `notes`.** It reports the coding and
      the complete cases, e.g. `Covariate Status (Live=…, Cryo=…): categorical
      coded Cryo=0, Cyro=1, Ext=2, Live=3, UTHSC=4, UTHSC2=5, 90 of 93
      samples have a value, 34 with both the trait and the covariate; 6
      levels`. Report that *n* next to any LOD/p change — a peak that shifts
      because 60 samples left the fit is not a discovery. Note the data's own
      typo: `Cyro` and `Cryo` are two levels.
    * **Categorical vs numeric.** Numeric text passes through as numbers;
      anything else with 2–20 distinct values among *the strains being
      mapped* is coded 0..k−1 in sorted order; more than 20, a single level,
      or a `map` that misses a value is a `422` asking for an explicit `map`.
      Measured on BXD (trait 10001, `BXD.4`): `RRID` is refused — `422 …
      82 distinct values, too many to code automatically` — while `Status`
      (6 levels) and `Epoch` are accepted.
    * **Compound values are a trap.** `SeqCvge` mixes numbers with two-library
      values (`40+31L`) and carries 30 distinct values across the group: if
      few enough fall inside the mapped panel it is accepted, but as a
      *categorical* — and the GEMMA run then fails (`502 gemma_failed`) rather
      than telling you why. The same is true of any mostly-numeric attribute
      with a few odd entries. Send the coding yourself:
      `{"case_attribute": "SeqCvge", "map": {"40+31L": 71, "40": 40, …}}`, or
      inline `{"name": …, "kind": "numerical", "values": […]}`.
    * **GEMMA** gets k−1 dummy columns (first level = reference) and *drops*
      a level none of the fitted individuals carries; the reply says so
      (`GEMMA fits 4 covariate column(s): Status.Ext, …` and `Covariate
      column(s) left out because they do not vary …`). R/qtl takes the single
      level column and dummy-expands it itself, so its note has no such line.
    * Missing covariate values drop those individuals from the fit (both
      engines), which `notes` reports as `Missing covariate values: …`.
    * At most 8 covariates, and they need a `group` (the individuals are
      aligned by name, which a bare `genotype_file` request does not carry).
  * `genotype_version`: list them with `/groups/{g}/genotypes` (BXD.8 is current).
* Timings seen: R/qtl about 7 s (1 MB reply), GEMMA about 3 s (1.9 MB), plots
  instant. They are synchronous; `?id=` only adds progress logging.
* Progress: `GET /compute/{rqtl,gemma}/status/{id}` returns `lines` and a
  `pointer` to pass back as `peek`; the last line is `COMPLETION_MARKER` or
  `FAILURE_MARKER: <error>`. It exists only for an `id` you passed on the POST
  (otherwise `404 run_not_found`).
* `/plot/*` accepts the whole compute reply or just its `data`; query params
  `trait`, `title`, and `bonferroni=false` (GEMMA only). The title names the
  trait from the reply's own `Trait: ...` note (both compute endpoints write
  it first), so posting the whole reply back is enough; `?trait=` wins over
  the note.
* Two traits really do draw two different plots (traces, peaks, annotation);
  the title is the one thing that was identical before the note existed.
* `/compute/correlation` is a stub with canned data. Do not use it.
* You never supply genotypes: the server uses its own files.

**PheWAS (GET)** `/groups/{group}/phewas?locus=<marker>` or `?chr=&mb=`,
`window` (bp, default 10000), `genotype_version`. Returns published traits whose
recorded peak marker lies in the window, strongest first. An empty list is a
real answer.

**Data (GET, all need the token)**

| path | gives you |
|---|---|
| `/species`, `/species/{s}`, `/species/{s}/datasets` | species, expression datasets |
| `/groups?species=`, `/groups/{g}`, `/groups/{g}/strains`, `/groups/{g}/datasets` | populations and samples |
| `/groups/{g}/case-attributes` | the group's metadata: `.data.names`, and `.data.descriptions` **saying what each one means** — read those before fitting a covariate |
| `/groups/{g}/traits`, `/groups/{g}/traits/{id}`, `/groups/{g}/traits/{id}/values` | published traits (the first is the bulk list) |
| `/groups/{g}/traits/{id}/case-attributes/{attribute}` | a trait's case-attribute values, in the trait's own sample order, plus `description` and a `distinct` count — the honest way to see whether an attribute is usable as a covariate (a constant one is not) |
| `/groups/{g}/genotypes[/{version}[/markers]]` | genotype versions and markers, and the marker names `/compute/fitqtl` takes |
| `POST /compute/{rqtl,gemma,fitqtl}` | mapping (POST, not a GET): the scan, the LMM, and effect sizes at named markers — all three take `covariates` and `normalize` |
| `/datasets/{ds}[/probesets[/{p}[/values]]]`, `/traits/{ds}[/{t}[/values]]` | expression data |
| `/publications/{pubmed_id}/traits` | traits a paper reports (PubMed id, not `publication_id`) |
| `/accessions/{acc}`, `/mapping-methods` | GN accession lookup, methods offered |

**Bulk jobs.** Fetch `/groups/{g}/traits` once (the whole PublishXRef list:
`id`, `phenotype_id`, `dataset`, `abbreviation`, `nsamples`; about 15 000 for
BXD), then `.../traits/{id}/values` per trait, a few at a time. Prefer one bulk
call to a thousand small ones; do not use `/search` for bulk work.

**Other examples:** `/datasets/Br_U_0803_M/probesets/100001_at` (one expression
trait), `/publications/2760876/traits` (PubMed id), and
`/search?q=cerebellum AND group:BXD&limit=5`.

## 3a. Gene link-outs: where else to send a user about a gene

GeneNetwork2's trait page links out to a dozen services by gene symbol. These
are the ones that still resolve (checked 2026-11-11, HTTP 200 and — where the
page is not a JavaScript shell — the gene actually found), with the URL template
to hand to a user. `{SYMBOL}` is the gene symbol; taxa are **mouse 10090, rat
10116, human 9606**.

| service | URL template | what it is for |
|---|---|---|
| **RGD** (gene search) | `https://rgd.mcw.edu/rgdweb/elasticResults.html?term={SYMBOL}&category=Gene&species={Mouse\|Rat\|Human}` | the best single hub *for mouse and rat*: orthologs, phenotypes, expression, QTLs. Works for all three species |
| **MGI** | `https://www.informatics.jax.org/searchtool/SearchResults?query={SYMBOL}` | mouse gene authority: alleles, phenotypes, knockouts |
| **Alliance of Genome Resources** | `https://www.alliancegenome.org/search?category=gene&q={SYMBOL}` | cross-species reference: orthologs and disease models |
| **NCBI Gene** | `https://www.ncbi.nlm.nih.gov/gene/?term={SYMBOL}[gene]+AND+mouse[orgn]` | the canonical record; also the source of Gene IDs |
| **Wikidata** | `https://www.wikidata.org/wiki/{QID}` | identifiers and aliases across databases. The QID is what `/genes/{SYMBOL}/aliases` returns, so this link costs nothing to build |
| **UniProt** | `https://www.uniprot.org/uniprotkb?query=gene:{SYMBOL}+AND+organism_id:{TAXON}` (or `…/uniprotkb/{ACCESSION}`) | protein entry, domains, variants. Search by symbol, link by accession |
| **Ensembl** | `https://www.ensembl.org/Mus_musculus/Gene/Summary?g={ENSEMBL_ID}` | genome browser, transcripts, orthologs. Needs the Ensembl gene id |
| **GTEx** | `https://www.gtexportal.org/home/gene/{HUMAN_SYMBOL}` | human tissue expression. **Human symbol only** — a mouse symbol shows an empty report |
| **STRING** | `https://string-db.org/cgi/network?identifier={SYMBOL}&species={TAXON}` | protein–protein interaction network |
| **GeneMANIA** | `https://genemania.org/search/{mouse\|rat\|human}/{SYMBOL}` | functional association network, per species |
| **Open Targets** | `https://platform.opentargets.org/search?q={SYMBOL}` | target–disease associations (human) |
| **Human Protein Atlas** | `https://www.proteinatlas.org/search/{SYMBOL}` | human tissue/cell expression at protein level |
| **EBI GWAS Catalog** | `https://www.ebi.ac.uk/gwas/search?query={SYMBOL}` | published GWAS associations |
| **BioGPS** | `http://biogps.org/?org=10090#goto=genereport&id={SYMBOL}` | mouse expression across tissues and datasets |
| **Allen Brain Atlas** | `https://mouse.brain-map.org/search/show?search_type=gene&search_term={SYMBOL}` | in-situ hybridisation in mouse brain |
| **GeneCup** | `https://genecup.org/progress?type=GWAS&query={SYMBOL}` | literature-mined gene–disease links |

**Do not offer these** (checked at the same time): HomoloGene (NCBI retired it
in 2021 — the URL answers with a notice), UniGene (retired, HTTP 410),
Genotation and GeneBridge (no DNS/connection), PhenoGen (404), the old chibi
GEMMA gene page, wiki-Pi. PANTHER answers 403 to a bot but works in a browser,
so offer it only as a "works in your browser" link:
`https://www.pantherdb.org/genes/geneList.do?searchType=basic&fieldName=all&organism=all&listType=1&fieldValue={SYMBOL}`.

**How to use them.** Quote the symbol and the species you found it in, and let
the user click. Do not synthesise what the other service says — say *"RGD has a
gene report for Shh (rat)"* and give the link, not a paraphrase of a page you
did not read. For a **rat** symbol prefer RGD and GeneMANIA(rat); for a **mouse**
symbol all of the above except Human Protein Atlas and GWAS apply; for a **human**
symbol use GTEx, Open Targets and the Protein Atlas.

**The API builds them for you.** `GET /genes/{SYMBOL}/aliases` returns a
`links` array (`[{name, href, description}]`) built from the table above for the
symbol and species it actually matched -- RGD, MGI, BioGPS and the Allen Brain
Atlas for a mouse gene; RGD, GeneMANIA and UniProt for a rat one. It also
returns `label`, the entity's canonical symbol in that species, which is how you
get the **human** symbol a human-only service needs:

    GET /genes/Shh/aliases?species=human
    → label "SHH", links […] including GTEx → https://www.gtexportal.org/home/gene/SHH

So a mouse `Shh` reaches GTEx through the human ortholog's label, and you never
hold a URL template yourself. `stale: true` means Wikidata was unreachable and
the cached entity was served -- the links still work, the label may be absent
from an older cache entry (retry and it refreshes).

## 4. Gotchas that cost time

* Compute replies are big (about 21 000 rows; GEMMA about 2 MB). Save to a file,
  inspect with `jq` / `python3 -I`. Piping to `head` forces a rerun.
* Rows are positional; names are in `meta.columns`.
* Not every value maps: trait 20010's 41 values map as 39, because C57BL/6J and
  DBA/2J are not in the BXD.8 genotype file, and `include_parents_f1: true` does
  not change that. `/groups/{g}/strains` flags such samples with
  `used_for_mapping: false`, which predicts it. `notes` on the reply says how many
  matched; tell the user.
* A peak is often a plateau: neighbouring markers share the same LOD / p-value
  (trait 20010's GEMMA maximum, 5.79, is shared by several markers from 182.6 to
  184.3 Mb). So the trait record's `locus`, R/qtl's peak, the plot summary and a
  reply's `next` can each name a different marker of the same strength. That is
  not an error: say "peak region" and use the marker the reply's `next` gives.
* **Before you believe a peak, count the animals in its genotype class.**
  `GET /groups/{g}/genotypes/{v}/markers/{marker}/classes?trait=<id>` gives
  `n`, `mean`, `sd`, `min`, `max` per class at that marker (no genotypes, no
  per-animal values). A peak carried by 1–2 animals is a different claim from
  one carried by a hundred, and the reply says which classes are thin. On DOL
  trait 10003 GEMMA's top marker `JAX00418132` (chr16 36.35 Mb, −log10 p 7.15)
  is `A n=2 mean 1.8`, `H n=20 mean 2.53`, `B n=171 mean 3.95`: the two A
  animals are the trait's two lowest values, and GEMMA's next two hits are the
  same pattern mirrored (`1 A` among 171, then `2 B` among 171). That block is
  a haplotype carried by a handful of extreme animals, not a population effect
  — say so rather than reporting 7.15 as a locus.
* **When R/qtl finds a peak R/qtl2 doesn't**, the model is the first suspect,
  not the data: R/qtl fits each marker's own calls (dropping the least
  frequent of three codes), R/qtl2 fits HMM probabilities with the H calls
  re-imputed. `{"genotype_model": "f2"}` keeps the het calls as a third
  genotype and shows what that recovers (BXD trait 16627: LOD 3.73 → 9.68 at
  `rs33299937`, against R/qtl's 9.59) — a diagnostic, not a result, since an
  inbred line has no heterozygotes. The reply says `Cross: riself` or
  `Cross: f2`: quote it.
* **`H` is dropped for R/qtl2 on a `riself` panel** (`@type:riset`: BXD, DOL,
  most RI sets), and kept by GEMMA as dosage 0.5. R/qtl2 writes those cells as
  missing and its HMM re-imputes them from flanking markers, so the animals are
  not lost — the observation is. A marker that is mostly `H` becomes nearly
  monomorphic for R/qtl2 (2 A vs 171 B at the marker above) and cannot produce
  a peak there, which is why the two engines can name different markers of the
  same region. Check `/frequencies` or `/classes` before calling one engine
  wrong.
* **GEMMA's `af` is not an allele frequency.** It is `mean(dosage)/2` under the
  BIMBAM coding the deployment ships (B=1, H=0.5, A=0), so `af 0.469` for
  `JAX00418132` is half the B-allele frequency 0.935, not the A-allele
  frequency 0.065. Take allele frequencies from
  `.../markers/{marker}/frequencies`. Likewise `allele1`/`allele0` in GEMMA's
  output are the BIMBAM `X, Y` placeholders, so the sign of `beta` is not a
  strain.
* **Do not compare scores across engines as if they were the same scale.**
  GEMMA reports −log10 p, R/qtl and R/qtl2 report LOD, qtlreaper reports LRS
  (`lod = lrs / 4.604`). The same region at "7.15" and "3.92" is not a
  disagreement, and an LMM's LOD is not a Haley-Knott LOD.
* PheWAS window matters: at the trait-20010 peak, 10 kb finds 4 traits and 1 Mb
  finds 19. The reply's `next` offers a ten-times wider window.
* Public-data rule: `public > 0` and `confidentiality < 1`, or older than five
  years; otherwise 403, do not retry. Metadata is visible even when values are not.
* There are no write endpoints, and the search index holds public documents only.
* Older docs said `POST /compute/*` is refused with 403. On this build it works.
  If you do get a 403 there, that is the answer: do not work around it. If the
  task needs private data or computation you are not granted, stop and ask a
  human: that needs an account and a grant, not a different URL.

## 5. Search

`/search?q=` is Xapian: juxtaposed terms are **OR**, so write `AND`; free text
is stemmed and case-insensitive; quoted phrases, wildcards and hyphenated words
match nothing. Fields: exact `group`, `dataset`, `species`, `author`, `tissue`,
`symbol`, `chr`, `peakchr`, `position`; stemmed `description`, `rif`, `wiki`;
ranges `mean`, `peak`, `peakmb`, `additive`, `year` (`field:lo..hi`).
Aliases: `pos` for `position`, `RIF` for `rif`, `WIKI` for `wiki`; `Hs`/`Mm`
are species-location shorthands. Values are lowercased, so case does not matter.
A prefix whose value is not in the index matches nothing; that is "no such
value", not an error. `NEAR`/`ADJ` do not work either. A dangling operator or
unbalanced quote is `422 invalid_query` with Xapian's reason in `detail`: fix
it, do not retry. `type=phenotype|gene`, `limit` up to 100, `page`. There is no
total: counting means paging, and each page costs time proportional to the
matches left, so do not page through thousands of hits. For bulk work use
`/groups/{g}/traits`.

`.data` is an array of hits. Each has `name` (the **trait id**, e.g. `"21614"`),
`dataset`, `dataset_fullname`, `group`, `species`, `authors`, `year`,
`description`, `mean` and more. Without `group:` you also get other groups'
phenotypes (for example `C57BL-6JxC57BL-6NJF2Publish`), whose ids only mean
something inside their own group; add `AND group:BXD` to stay in BXD.

## 6. Tokens and errors

A GeneNetwork team member issues the token by hand; it cannot be self-served.
If you have none, ask: "To read GeneNetwork data I need a GeneNetwork API token
(a JWT) for https://vesuvio.pu.ac.ke/gn4. It is issued by hand; it helps to say
who it is for, which groups/datasets, how long (default 30 days) and what for."
Ask the user to pass it out of band, not in a public issue, a commit or a
shared chat. Send it as `Authorization: Bearer <token>` on every request except
`/version` and `/openapi.json`, and keep it for the whole session. Never put it
in a URL, a commit, a log line or anything shown to someone else, and do not
hand it to other users or paste it into shared state.

| status | do |
|---|---|
| 401 `invalid_token` | `detail` says `ExpiredSignature` (expired) or `InvalidIssuer` (wrong deployment). You cannot renew it: tell the user, ask for a fresh one quoting the `request_id`, retry **once**, then stop and report; do not loop or look for another credential |
| 403 `forbidden` | not public: do not retry |
| 404 `not_found` | check the name; `next` lists valid values |
| 422 `invalid_query`, `invalid_lmm` | fix the query or parameter; do not retry |
| 429 | back off and honour `Retry-After` (not emitted today, but in the contract) |
| 502/503 | transient: back off, max about 5 tries |

Errors are RFC 7807-style JSON (`type`, `title`, `status`, `code`, `detail`,
`operation_id`, `spec_url`, `request_id`, sometimes `next`); `type`, `spec_url`
and `next` are absolute paths under `/gn4/`. Quote `request_id` when reporting
a problem (also in the `x-request-id` header). No rate limit is configured, but
be considerate: a few concurrent requests, no tight polling loops.

## 7. Keep this skill current: reload it if GeneNetwork changes mid-session

Do not rely on a version written into this file. Instead, **remember the build
you saw at the start of the session** and compare against it later.

1. At the start (section 0), note `curl -s https://vesuvio.pu.ac.ke/gn4/version
   | jq -r .data.git` as your session baseline.
2. Check `/version` again when something behaves differently from what is
   written here (an unexpected 404, a new field in `actions`, a path in
   `openapi.json` this file does not mention), and before a long or repeated
   piece of work.
3. If the revision is no longer the baseline, GeneNetwork was updated while you
   worked. **Suggest to the user that you reload SKILL.md from the repo**
   (`docs/www/SKILL.md` in `https://git.genenetwork.org/genenetwork4`; a sparse
   clone of `docs` is enough), and use that copy from then on. One line is
   enough, for example: "GeneNetwork was updated since we started (<old> ->
   <new>); I'd reload SKILL.md from docs/www/ to pick up any changes. Shall I?"
   Then set the new revision as your baseline.
4. If the repo is unreachable from the sandbox, fall back to
   `/gn4/openapi.json`, which always matches the running server, and tell the
   user the skill file itself could not be refreshed.

## 7a. Project memory: the answers this repo has already worked out

This repository carries an `ai-memory` store (`ai-memory.sqlite`) linking each
commit to the reasoning behind it, plus an optional Xapian topic index over
both. A previous session's context window is gone; this is what survives it,
and it is cheaper than re-deriving anything:

```sh
ai-memory -C genenetwork4 search 'pve'          # SQLite FTS over the reasoning: ~15 ms
ai-memory -C genenetwork4 why gemma.compute_gemma   # commits touching a symbol, with its reasoning
ai-memory -C genenetwork4 --index DIR search 'topic:pve repo:genenetwork4'   # topic index: ~100 ms
```

`topic:` terms come from the session id, so `topic:ravanan-slurm`,
`topic:reference-genomes`, `topic:gemma-pve` are the entry points that exist.

Two things to look for before starting work that touches an engine, a cache or
a cluster:

- **a RUNBOOK entry** — the operational ones carry the actual commands, ports
  and fingerprints, e.g. bringing up the Slurm container from `gn-machines` and
  testing `/compute/gemma/ravanan` against it, with the four prerequisites that
  were needed the first time (a ravanan build with the `--slurm-api-endpoint`
  fix, `ccwl` on `PATH`, the server's `HOME` writable inside the container,
  `GUIX_PACKAGE_PATH` for the manifest);
- **a "why is this like this" entry** — several endpoints here exist because
  something else was wrong (`-pve` reporting, the `next`-link graph, the
  reference-genome linkage). Reading it is faster than deciding they are odd.

The store is per repository and local (gitignored): it survives a cleared
session, not a lost machine, which is why the durable version of every
conclusion is also in a commit message and in these docs.

## 8. Reference docs

The docs live in the public repo and are the authoritative version:

```sh
git clone --depth 1 --sparse https://git.genenetwork.org/genenetwork4
cd genenetwork4 && git sparse-checkout set docs
```

About 2.5 MB; drop `--sparse` for the whole tree. Then read:

| file | what it is |
|---|---|
| `docs/www/SKILL.md` | this page |
| `docs/search-engines.md` | how `/search` works, engines, quirks |
| `README.md` | every endpoint, parameter and response shape |
| `docs/nginx.md`, `docs/www/operator-checklist.md` | operator-level; you do not need them |

**Clone; do not fetch the web UI.** `git.genenetwork.org/.../tree/...` pages sit
behind a proof-of-work challenge (Anubis) and the raw endpoint answers 403, so
a plain fetch returns an HTML challenge page, not Markdown. If a fetch comes
back as HTML mentioning Anubis, stop, do not retry, and use the copy you have.
The git protocol itself passes. Sandboxes sometimes cannot reach it either;
`/gn4/openapi.json` is the fallback.

To compare revisions, `git -C genenetwork4 log -1 --format=%h` against `.data.git`
from `/version`: they need not match exactly (docs can be newer than the
running build), but a large gap means the server may lack what the docs describe.