about summary refs log tree commit diff
path: root/docs/genotypes.md
blob: 55f7a6df20c27b557cf4f54944ac542229bcbc54 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
# Genotype files: where they are, which one is current, how to map with them

`/compute/rqtl` and `/compute/gemma` take paths that exist on the
server, so whoever calls them has to know two things the API does not
tell them: where the deployment keeps genotype files, and **which**
version of a panel's genotypes is the current one. Both are answered by
long-standing GeneNetwork convention, reproduced here because it is not
discoverable from this repository's code.

The deployment's copy is `/home/wrk/tmp/ai-root/factory/gn4-data/
genotype-files`, laid out like GeneNetwork2's own `genotype_files`:

```
genotype/<Group>.geno                 GN genotype format (Chr/Locus/cM/Mb + one column per individual)
genotype/<Group>.json                 per-group metadata: the list of genotype files, newest first
genotype/bimbam/<file>_geno.txt       GEMMA/BIMBAM genotype (0/0.5/1) + <file>_snps.txt + <file>_pheno.txt
genotype/rqtl2/<file>_{geno,gmap,pheno}.csv, <file>.json    R/qtl2 bundle
genotype/json/<Group>.json            older marker dump (not the version list)
mapping/<Group>.bed|.bim|.fam         PLINK, with covariates in mapping/
```

## Which file is current

GeneNetwork2 (`gn2/base/data_set/datasetgroup.py:get_genofiles`,
`gn2/base/webqtlConfig.py:GENODIR`): read `genotype/<Group>.json` and
take its `genofile` key, which is a **list, newest first**, each entry
`{title, location, sample_list?}`. The mapping UI offers the list and
defaults to the first. With no JSON, it falls back to `<Group>.geno`.
The user's choice rides along in `genofile_string` as
`<file>:<title>` (`display_mapping_results.py`).

So "the latest version" is `genofile[0]`, not the highest number and not
the newest mtime. For BXD that is:

```json
{"title": "WGS-based (Mar2022)", "location": "BXD.8.geno", "sample_list": [...235 names...]}
```

followed by `BXD.geno` ("Array-based 2017–2021"), `BXD.4.geno`
("Classic 2001-2016") and `BXD.2.geno` ("Test"). Note `BXD.8.geno` is
not even in the `GenoFile` table — that table is older and only ever had
the classic/array files — so the JSON, not MariaDB, is the authority on
which version is current. `GenoFreeze` is one row per group
("BXD Genotypes", 2003), not one per version.

## How each engine is fed

- **GEMMA** (GN2 `marker_regression/gemma_mapping.py`): strip `.geno`
  from the location and use
  `genotype/bimbam/<name>_geno.txt` with `<name>_snps.txt` — for BXD's
  current file that is `bimbam/BXD.8_geno.txt` +
  `bimbam/BXD.8_snps.txt` (21 056 markers × 235 individuals). The `.geno`
  file itself is *not* what GEMMA gets.
- **R/qtl** (GN2 `marker_regression/rqtl_mapping.py`): pass
  `genotype/<location>` straight through — `genotype/BXD.8.geno` — and
  let `scripts/rqtl_wrapper.R`'s `geno_to_csvr` read the GN format.
- **The sample list** (GN3 `gn3/db/genotypes.py:load_genotype_samples`,
  `gn3/db/datasets.py`): the first non-`#`/`@` line of the `.geno` file
  is the header; if its fourth column is `Mb`, sample names start at
  column 5, otherwise at column 4. That is the only authoritative sample
  order, and the only place it is stored.

## The alignment rule, which the API now applies

Both engines match values to individuals **by position**: R/qtl's cross
format has no per-sample names, and `write_pheno_file` (src/gemma.rs)
writes bare values. Inside the API that no longer leaks out: a request
names a **group** (and optionally a version), passes the trait's values
as named `samples`, and `src/genotypes.rs` aligns them onto the panel by
name, dropping names the panel does not have and filling `null` for
individuals the trait does not cover. Parents and F1s are excluded unless
`include_parents_f1` is set — and excluded the only way that works, by
subsetting the genotype matrix (a cached panel-subset file under the
engine's cache directory), because null-filling keeps an individual out
of the fit but not out of GEMMA's kinship matrix or R/qtl's cross.

`scripts/rqtl_wrapper.R` still refuses a count mismatch, which is what
catches a hand-built request that bypasses the alignment (an explicit
`genotype_file` with the wrong number of `samples`).

## Which engine a group offers

`InbredSet.MappingMethodId` (GN2 `get_mapping_methods`): 1 → GEMMA,
QTLReaper and R/qtl; 2 → GEMMA; 3 → R/qtl; 4 → GEMMA and PLINK. On this
database: `BXD` = 1, `HET3-ITP` = 3, `HSNIH-Palmer` = 2, `HS` = NULL.

`HET3-ITP` is UM-HET3, a 4-way cross (`@type:4-way`, `@mat:AB`,
`@pat:CD` in its `.geno` header), so R/qtl is the right engine even
though `bimbam/HET3-ITP.*` files exist — those are a biallelic
projection of a four-founder panel.

## Measured runs (2026-11-07, this deployment)

Values came from `/groups/{group}/traits/{id}/values`, were re-indexed
onto the genotype file's sample list by name with `null` for the rest,
and were posted to the API.

| panel / genotype file | engine | trait | panel | trait values | time | result |
|---|---|---|---|---|---|---|
| BXD, `BXD.8.geno` (WGS Mar2022) | GEMMA (bimbam) | 10001 cerebellum weight | 235 | 34 | 17.1 s | 21 055 marker rows |
| BXD, `BXD.8.geno` | R/qtl scanone | 10001 | 235 | 34 | 7.5 s | 21 056 markers |
| HET3-ITP, `HET3-ITP.12.geno` (Apr 2023) | R/qtl scanone | 10000 lifespan | 6 447 | 6 438 | 83.1 s | 893 markers, peak LOD 7.95 at `1_24042124` (chr 1, 24.04 Mb) |

## What the API tells a caller about a genotype file

Three routes, all behind the bearer token like every other endpoint:

| route | what it gives |
|---|---|
| `GET /groups/{group}/genotypes` | versions newest-first, each with its `file` name, title, `individuals`, `markers`, which engines can use it, and which is `current` |
| `GET /groups/{group}/genotypes/{version}` | that file's `file` name, its header `@`-labels, the panel's `individuals` in file order, the `strains` a mapping uses, counts, and per-chromosome marker counts |
| `GET /groups/{group}/genotypes/{version}/markers?page=&limit=&chr=` | the marker table: name, chromosome, cM, Mb — paged, positional rows |

**The genotype calls are never served.** A caller sees the panel, the
markers and the header, which is what it needs to frame a mapping and to
check one; the matrix itself stays on the server, and the compute
endpoints take a group and a version rather than a path.

## Reproducing the measurements

The measurements above were taken before the API took over the
resolution, by doing its steps by hand. Now it is two calls:

```sh
# what can BXD be mapped with, and which version is current?
curl -s -H "Authorization: Bearer $TOKEN" "$API/groups/BXD/genotypes"

# map a trait over it: group + the trait's own values, no paths
curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -X POST "$API/compute/rqtl" -d '{
    "group": "BXD",
    "trait_name": "BXDPublish::10001",
    "samples": [{"name": "BXD1", "value": 61.4}, ...]
  }'
```

Add `"genotype_version": "BXD.4"` to map an older panel. The response's
`notes` names the resolved version, the panel size after excluding
parents/F1s, and how many supplied values aligned — that line is what
makes a run reproducible without publishing paths.

## Parents and F1s are excluded by default

GeneNetwork2 and GeneNetwork3 map **without** the parents and F1s unless
asked otherwise: GN2's `DatasetGroup.incparentsf1` defaults to `False`
(`gn2/base/data_set/datasetgroup.py`) and its `ParInfo` parents are only
added to the genotype object when it is on; GN3 spells the same thing as
`get_sample_list(group, inc_par=False, inc_f1=False)`
(`gn3/db/datasets.py`). The parent/F1 lists themselves come from
`parents_and_f1s.json` in the data directory.

On this deployment the distinction is narrower than it sounds, because
the panels already tell you what they hold:

- `BXD.8.geno` is 235 columns = **198 RI strains + 37 F1s**, and **no
  parent columns at all** (C57BL/6J and DBA/2J are not in the file; the
  JSON's 235-entry `sample_list` is the same 198+37). So "exclude
  parents and F1s" here means "use the 198 strains".
- `HET3-ITP.12.geno` is 6 447 **individual animals** (1 953 `JL…` plus
  4 494 `UM…`), no parents, no F1s — UM-HET3 maps individuals, so the
  default is moot.

The first BXD mappings above passed all 235 columns with `null` for the
F1s (and for every strain the trait does not cover), i.e. the *phenotype*
set was the trait's 34 strains while the *genotype* side spanned 235. That
is not GN2's default, and it changes GEMMA's kinship matrix (computed over
genotypes, not phenotypes) and R/qtl's cross. Measured on trait 10001
both ways, it happened not to change the answer:

| columns passed | GEMMA best | R/qtl best LOD |
|---|---|---|
| all 235 (strains + F1s) | p 5.9e-07 at `rsm10000005700`, chr 8 | 2.9320 at `rs48756159`, chr 8 |
| 198 strains only | p 5.9e-07 at `rsm10000005700`, chr 8 | 2.9320 at `rs48756159`, chr 8 |

So `null`-filling is enough to keep the *phenotypes* right, but not to
exclude an individual from the model: the genotype matrix has to be
subset. When the API takes over the alignment, it should subset to the
strains by default and offer "include parents and F1s" explicitly, which
is what GN2's checkbox does.

## No restart is needed to use a new genotype file

The path travels in each request and nothing in the server caches a
panel, so pointing a request at a newer file (`BXD.8.geno` instead of
`BXD.geno`, `HET3-ITP.12.geno` instead of `.11`) takes effect on the next
request. Both engines' result caches key on file **contents**, not paths:
`src/rqtl.rs` reads the genotype file and hashes it into the output
filename, and gemma-wrapper names its cached kinship by a content hash --
so a file that has changed is recomputed, not silently reused.

What does need a restart:

- a new binary (any Rust change), and `GENENETWORK4_TEST_TOKEN`, which is
  decided when the router is built;
- the process environment: `PATH` and `R_LIBS_SITE` (R/qtl's `Rscript`
  and its `qtl`/`optparse`/`stringi`/`stringr` packages), and the values
  the process fixes at first use -- `SQL_URI` (the MySQL pool),
  `XAPIAN_DB_PATH`/`SEARCH_ENGINE` (the open index handle);
- not `scripts/rqtl_wrapper.R`: it runs as a subprocess per request, so
  editing it takes effect immediately.

## What is still messy

`HET3-ITP` has eleven `.geno` versions on disk (`.2` … `.12`) and **no
`genotype/HET3-ITP.json`**, so there is no recorded list and no recorded
"current" — `.12` (6447 individuals, 894 markers, Apr 2023) is the newest by both
numbering and date -- the `rqtl2/` bundles, for what it is worth, only
exist for `.2`, `.3` and `.4`, so they do not mark the current version
either -- but that is inference, not a documented default. Its traits
are per **animal**, not per strain (lifespan: 23 879 values over
several cohorts), so only 6 438 of them fit the `.12` panel; the rest
belong to cohorts that genotype file does not contain. Mapping them
"latest version" therefore silently drops cohorts — correct per the
file, wrong as biology until the version/cohort pairing is recorded.

The fix on the GN4 side is small and worth doing: align trait values
onto the panel inside the compute handlers (by name, from the `.geno`
header), so a caller passes the values it has and the server does the
re-indexing. That also makes the `actions` bodies directly usable.