# Genotype files: where they are, which one is current, how to map with them `/compute/rqtl` and `/compute/gemma` take paths that exist on the server, so whoever calls them has to know two things the API does not tell them: where the deployment keeps genotype files, and **which** version of a panel's genotypes is the current one. Both are answered by long-standing GeneNetwork convention, reproduced here because it is not discoverable from this repository's code. The deployment's copy is `/home/wrk/tmp/ai-root/factory/gn4-data/ genotype-files`, laid out like GeneNetwork2's own `genotype_files`: ``` genotype/.geno GN genotype format (Chr/Locus/cM/Mb + one column per individual) genotype/.json per-group metadata: the list of genotype files, newest first genotype/bimbam/_geno.txt GEMMA/BIMBAM genotype (0/0.5/1) + _snps.txt + _pheno.txt genotype/rqtl2/_{geno,gmap,pheno}.csv, .json R/qtl2 bundle genotype/json/.json older marker dump (not the version list) mapping/.bed|.bim|.fam PLINK, with covariates in mapping/ ``` ## Which file is current GeneNetwork2 (`gn2/base/data_set/datasetgroup.py:get_genofiles`, `gn2/base/webqtlConfig.py:GENODIR`): read `genotype/.json` and take its `genofile` key, which is a **list, newest first**, each entry `{title, location, sample_list?}`. The mapping UI offers the list and defaults to the first. With no JSON, it falls back to `.geno`. The user's choice rides along in `genofile_string` as `:` (`display_mapping_results.py`). So "the latest version" is `genofile[0]`, not the highest number and not the newest mtime. For BXD that is: ```json {"title": "WGS-based (Mar2022)", "location": "BXD.8.geno", "sample_list": [...235 names...]} ``` followed by `BXD.geno` ("Array-based 2017–2021"), `BXD.4.geno` ("Classic 2001-2016") and `BXD.2.geno` ("Test"). Note `BXD.8.geno` is not even in the `GenoFile` table — that table is older and only ever had the classic/array files — so the JSON, not MariaDB, is the authority on which version is current. `GenoFreeze` is one row per group ("BXD Genotypes", 2003), not one per version. ## How each engine is fed - **GEMMA** (GN2 `marker_regression/gemma_mapping.py`): strip `.geno` from the location and use `genotype/bimbam/<name>_geno.txt` with `<name>_snps.txt` — for BXD's current file that is `bimbam/BXD.8_geno.txt` + `bimbam/BXD.8_snps.txt` (21 056 markers × 235 individuals). The `.geno` file itself is *not* what GEMMA gets. - **R/qtl** (GN2 `marker_regression/rqtl_mapping.py`): pass `genotype/<location>` straight through — `genotype/BXD.8.geno` — and let `scripts/rqtl_wrapper.R`'s `geno_to_csvr` read the GN format. - **The sample list** (GN3 `gn3/db/genotypes.py:load_genotype_samples`, `gn3/db/datasets.py`): the first non-`#`/`@` line of the `.geno` file is the header; if its fourth column is `Mb`, sample names start at column 5, otherwise at column 4. That is the only authoritative sample order, and the only place it is stored. ## The alignment rule, which the API now applies Both engines match values to individuals **by position**: R/qtl's cross format has no per-sample names, and `write_pheno_file` (src/gemma.rs) writes bare values. Inside the API that no longer leaks out: a request names a **group** (and optionally a version), passes the trait's values as named `samples`, and `src/genotypes.rs` aligns them onto the panel by name, dropping names the panel does not have and filling `null` for individuals the trait does not cover. Parents and F1s are excluded unless `include_parents_f1` is set — and excluded the only way that works, by subsetting the genotype matrix (a cached panel-subset file under the engine's cache directory), because null-filling keeps an individual out of the fit but not out of GEMMA's kinship matrix or R/qtl's cross. `scripts/rqtl_wrapper.R` still refuses a count mismatch, which is what catches a hand-built request that bypasses the alignment (an explicit `genotype_file` with the wrong number of `samples`). ## Which engine a group offers `InbredSet.MappingMethodId` (GN2 `get_mapping_methods`): 1 → GEMMA, QTLReaper and R/qtl; 2 → GEMMA; 3 → R/qtl; 4 → GEMMA and PLINK. On this database: `BXD` = 1, `HET3-ITP` = 3, `HSNIH-Palmer` = 2, `HS` = NULL. `HET3-ITP` is UM-HET3, a 4-way cross (`@type:4-way`, `@mat:AB`, `@pat:CD` in its `.geno` header), so R/qtl is the right engine even though `bimbam/HET3-ITP.*` files exist — those are a biallelic projection of a four-founder panel. ## Measured runs (2026-11-07, this deployment) Values came from `/groups/{group}/traits/{id}/values`, were re-indexed onto the genotype file's sample list by name with `null` for the rest, and were posted to the API. | panel / genotype file | engine | trait | panel | trait values | time | result | |---|---|---|---|---|---|---| | BXD, `BXD.8.geno` (WGS Mar2022) | GEMMA (bimbam) | 10001 cerebellum weight | 235 | 34 | 17.1 s | 21 055 marker rows | | BXD, `BXD.8.geno` | R/qtl scanone | 10001 | 235 | 34 | 7.5 s | 21 056 markers | | HET3-ITP, `HET3-ITP.12.geno` (Apr 2023) | R/qtl scanone | 10000 lifespan | 6 447 | 6 438 | 83.1 s | 893 markers, peak LOD 7.95 at `1_24042124` (chr 1, 24.04 Mb) | ## What the API tells a caller about a genotype file Three routes, all behind the bearer token like every other endpoint: | route | what it gives | |---|---| | `GET /groups/{group}/genotypes` | versions newest-first, each with its `file` name, title, `individuals`, `markers`, which engines can use it, and which is `current` | | `GET /groups/{group}/genotypes/{version}` | that file's `file` name, its header `@`-labels, the panel's `individuals` in file order, the `strains` a mapping uses, counts, and per-chromosome marker counts | | `GET /groups/{group}/genotypes/{version}/markers?page=&limit=&chr=` | the marker table: name, chromosome, cM, Mb — paged, positional rows | **The genotype calls are never served.** A caller sees the panel, the markers and the header, which is what it needs to frame a mapping and to check one; the matrix itself stays on the server, and the compute endpoints take a group and a version rather than a path. ## Reproducing the measurements The measurements above were taken before the API took over the resolution, by doing its steps by hand. Now it is two calls: ```sh # what can BXD be mapped with, and which version is current? curl -s -H "Authorization: Bearer $TOKEN" "$API/groups/BXD/genotypes" # map a trait over it: group + the trait's own values, no paths curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ -X POST "$API/compute/rqtl" -d '{ "group": "BXD", "trait_name": "BXDPublish::10001", "samples": [{"name": "BXD1", "value": 61.4}, ...] }' ``` Add `"genotype_version": "BXD.4"` to map an older panel. The response's `notes` names the resolved version, the panel size after excluding parents/F1s, and how many supplied values aligned — that line is what makes a run reproducible without publishing paths. ## Parents and F1s are excluded by default GeneNetwork2 and GeneNetwork3 map **without** the parents and F1s unless asked otherwise: GN2's `DatasetGroup.incparentsf1` defaults to `False` (`gn2/base/data_set/datasetgroup.py`) and its `ParInfo` parents are only added to the genotype object when it is on; GN3 spells the same thing as `get_sample_list(group, inc_par=False, inc_f1=False)` (`gn3/db/datasets.py`). The parent/F1 lists themselves come from `parents_and_f1s.json` in the data directory. On this deployment the distinction is narrower than it sounds, because the panels already tell you what they hold: - `BXD.8.geno` is 235 columns = **198 RI strains + 37 F1s**, and **no parent columns at all** (C57BL/6J and DBA/2J are not in the file; the JSON's 235-entry `sample_list` is the same 198+37). So "exclude parents and F1s" here means "use the 198 strains". - `HET3-ITP.12.geno` is 6 447 **individual animals** (1 953 `JL…` plus 4 494 `UM…`), no parents, no F1s — UM-HET3 maps individuals, so the default is moot. The first BXD mappings above passed all 235 columns with `null` for the F1s (and for every strain the trait does not cover), i.e. the *phenotype* set was the trait's 34 strains while the *genotype* side spanned 235. That is not GN2's default, and it changes GEMMA's kinship matrix (computed over genotypes, not phenotypes) and R/qtl's cross. Measured on trait 10001 both ways, it happened not to change the answer: | columns passed | GEMMA best | R/qtl best LOD | |---|---|---| | all 235 (strains + F1s) | p 5.9e-07 at `rsm10000005700`, chr 8 | 2.9320 at `rs48756159`, chr 8 | | 198 strains only | p 5.9e-07 at `rsm10000005700`, chr 8 | 2.9320 at `rs48756159`, chr 8 | So `null`-filling is enough to keep the *phenotypes* right, but not to exclude an individual from the model: the genotype matrix has to be subset. When the API takes over the alignment, it should subset to the strains by default and offer "include parents and F1s" explicitly, which is what GN2's checkbox does. ## No restart is needed to use a new genotype file The path travels in each request and nothing in the server caches a panel, so pointing a request at a newer file (`BXD.8.geno` instead of `BXD.geno`, `HET3-ITP.12.geno` instead of `.11`) takes effect on the next request. Both engines' result caches key on file **contents**, not paths: `src/rqtl.rs` reads the genotype file and hashes it into the output filename, and gemma-wrapper names its cached kinship by a content hash -- so a file that has changed is recomputed, not silently reused. What does need a restart: - a new binary (any Rust change), and `GENENETWORK4_TEST_TOKEN`, which is decided when the router is built; - the process environment: `PATH` and `R_LIBS_SITE` (R/qtl's `Rscript` and its `qtl`/`optparse`/`stringi`/`stringr` packages), and the values the process fixes at first use -- `SQL_URI` (the MySQL pool), `XAPIAN_DB_PATH`/`SEARCH_ENGINE` (the open index handle); - not `scripts/rqtl_wrapper.R`: it runs as a subprocess per request, so editing it takes effect immediately. ## What is still messy `HET3-ITP` has eleven `.geno` versions on disk (`.2` … `.12`) and **no `genotype/HET3-ITP.json`**, so there is no recorded list and no recorded "current" — `.12` (6447 individuals, 894 markers, Apr 2023) is the newest by both numbering and date -- the `rqtl2/` bundles, for what it is worth, only exist for `.2`, `.3` and `.4`, so they do not mark the current version either -- but that is inference, not a documented default. Its traits are per **animal**, not per strain (lifespan: 23 879 values over several cohorts), so only 6 438 of them fit the `.12` panel; the rest belong to cohorts that genotype file does not contain. Mapping them "latest version" therefore silently drops cohorts — correct per the file, wrong as biology until the version/cohort pairing is recorded. The fix on the GN4 side is small and worth doing: align trait values onto the panel inside the compute handlers (by name, from the `.geno` header), so a caller passes the values it has and the server does the re-indexing. That also makes the `actions` bodies directly usable.