Genotype files: where they are, which one is current, how to map with them
/compute/rqtl and /compute/gemma take paths that exist on the
server, so whoever calls them has to know two things the API does not
tell them: where the deployment keeps genotype files, and which
version of a panel's genotypes is the current one. Both are answered by
long-standing GeneNetwork convention, reproduced here because it is not
discoverable from this repository's code.
The deployment's copy is /home/wrk/tmp/ai-root/factory/gn4-data/
genotype-files, laid out like GeneNetwork2's own genotype_files:
genotype/<Group>.geno GN genotype format (Chr/Locus/cM/Mb + one column per individual)
genotype/<Group>.json per-group metadata: the list of genotype files, newest first
genotype/bimbam/<file>_geno.txt GEMMA/BIMBAM genotype (0/0.5/1) + <file>_snps.txt + <file>_pheno.txt
genotype/rqtl2/<file>_{geno,gmap,pheno}.csv, <file>.json R/qtl2 bundle
genotype/json/<Group>.json older marker dump (not the version list)
mapping/<Group>.bed|.bim|.fam PLINK, with covariates in mapping/
Which file is current
GeneNetwork2 (gn2/base/data_set/datasetgroup.py:get_genofiles,
gn2/base/webqtlConfig.py:GENODIR): read genotype/<Group>.json and
take its genofile key, which is a list, newest first, each entry
{title, location, sample_list?}. The mapping UI offers the list and
defaults to the first. With no JSON, it falls back to <Group>.geno.
The user's choice rides along in genofile_string as
<file>:<title> (display_mapping_results.py).
So "the latest version" is genofile[0], not the highest number and not
the newest mtime. For BXD that is:
{"title": "WGS-based (Mar2022)", "location": "BXD.8.geno", "sample_list": [...235 names...]}
followed by BXD.geno ("Array-based 2017–2021"), BXD.4.geno
("Classic 2001-2016") and BXD.2.geno ("Test"). Note BXD.8.geno is
not even in the GenoFile table — that table is older and only ever had
the classic/array files — so the JSON, not MariaDB, is the authority on
which version is current. GenoFreeze is one row per group
("BXD Genotypes", 2003), not one per version.
How each engine is fed
- GEMMA (GN2
marker_regression/gemma_mapping.py): strip.genofrom the location and usegenotype/bimbam/<name>_geno.txtwith<name>_snps.txt— for BXD's current file that isbimbam/BXD.8_geno.txt+bimbam/BXD.8_snps.txt(21 056 markers × 235 individuals). The.genofile itself is not what GEMMA gets. - R/qtl (GN2
marker_regression/rqtl_mapping.py): passgenotype/<location>straight through —genotype/BXD.8.geno— and letscripts/rqtl_wrapper.R'sgeno_to_csvrread the GN format. - The sample list (GN3
gn3/db/genotypes.py:load_genotype_samples,gn3/db/datasets.py): the first non-#/@line of the.genofile is the header; if its fourth column isMb, sample names start at column 5, otherwise at column 4. That is the only authoritative sample order, and the only place it is stored.
The alignment rule, which the API now applies
Both engines match values to individuals by position: R/qtl's cross
format has no per-sample names, and write_pheno_file (src/gemma.rs)
writes bare values. Inside the API that no longer leaks out: a request
names a group (and optionally a version), passes the trait's values
as named samples, and src/genotypes.rs aligns them onto the panel by
name, dropping names the panel does not have and filling null for
individuals the trait does not cover. Parents and F1s are excluded unless
include_parents_f1 is set — and excluded the only way that works, by
subsetting the genotype matrix (a cached panel-subset file under the
engine's cache directory), because null-filling keeps an individual out
of the fit but not out of GEMMA's kinship matrix or R/qtl's cross.
scripts/rqtl_wrapper.R still refuses a count mismatch, which is what
catches a hand-built request that bypasses the alignment (an explicit
genotype_file with the wrong number of samples).
Which engine a group offers
InbredSet.MappingMethodId (GN2 get_mapping_methods): 1 → GEMMA,
QTLReaper and R/qtl; 2 → GEMMA; 3 → R/qtl; 4 → GEMMA and PLINK. On this
database: BXD = 1, HET3-ITP = 3, HSNIH-Palmer = 2, HS = NULL.
HET3-ITP is UM-HET3, a 4-way cross (@type:4-way, @mat:AB,
@pat:CD in its .geno header), so R/qtl is the right engine even
though bimbam/HET3-ITP.* files exist — those are a biallelic
projection of a four-founder panel.
Measured runs (2026-11-07, this deployment)
Values came from /groups/{group}/traits/{id}/values, were re-indexed
onto the genotype file's sample list by name with null for the rest,
and were posted to the API.
| panel / genotype file | engine | trait | panel | trait values | time | result |
|---|---|---|---|---|---|---|
BXD, BXD.8.geno (WGS Mar2022) |
GEMMA (bimbam) | 10001 cerebellum weight | 235 | 34 | 17.1 s | 21 055 marker rows |
BXD, BXD.8.geno |
R/qtl scanone | 10001 | 235 | 34 | 7.5 s | 21 056 markers |
HET3-ITP, HET3-ITP.12.geno (Apr 2023) |
R/qtl scanone | 10000 lifespan | 6 447 | 6 438 | 83.1 s | 893 markers, peak LOD 7.95 at 1_24042124 (chr 1, 24.04 Mb) |
What the API tells a caller about a genotype file
Three routes, all behind the bearer token like every other endpoint:
| route | what it gives |
|---|---|
GET /groups/{group}/genotypes |
versions newest-first, each with its file name, title, individuals, markers, which engines can use it, and which is current |
GET /groups/{group}/genotypes/{version} |
that file's file name, its header @-labels, the panel's individuals in file order, the strains a mapping uses, counts, and per-chromosome marker counts |
GET /groups/{group}/genotypes/{version}/markers?page=&limit=&chr= |
the marker table: name, chromosome, cM, Mb — paged, positional rows |
The genotype calls are never served. A caller sees the panel, the markers and the header, which is what it needs to frame a mapping and to check one; the matrix itself stays on the server, and the compute endpoints take a group and a version rather than a path.
Reproducing the measurements
The measurements above were taken before the API took over the resolution, by doing its steps by hand. Now it is two calls:
# what can BXD be mapped with, and which version is current?
curl -s -H "Authorization: Bearer $TOKEN" "$API/groups/BXD/genotypes"
# map a trait over it: group + the trait's own values, no paths
curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-X POST "$API/compute/rqtl" -d '{
"group": "BXD",
"trait_name": "BXDPublish::10001",
"samples": [{"name": "BXD1", "value": 61.4}, ...]
}'
Add "genotype_version": "BXD.4" to map an older panel. The response's
notes names the resolved version, the panel size after excluding
parents/F1s, and how many supplied values aligned — that line is what
makes a run reproducible without publishing paths.
Parents and F1s are excluded by default
GeneNetwork2 and GeneNetwork3 map without the parents and F1s unless
asked otherwise: GN2's DatasetGroup.incparentsf1 defaults to False
(gn2/base/data_set/datasetgroup.py) and its ParInfo parents are only
added to the genotype object when it is on; GN3 spells the same thing as
get_sample_list(group, inc_par=False, inc_f1=False)
(gn3/db/datasets.py). The parent/F1 lists themselves come from
parents_and_f1s.json in the data directory.
On this deployment the distinction is narrower than it sounds, because the panels already tell you what they hold:
BXD.8.genois 235 columns = 198 RI strains + 37 F1s, and no parent columns at all (C57BL/6J and DBA/2J are not in the file; the JSON's 235-entrysample_listis the same 198+37). So "exclude parents and F1s" here means "use the 198 strains".HET3-ITP.12.genois 6 447 individual animals (1 953JL…plus 4 494UM…), no parents, no F1s — UM-HET3 maps individuals, so the default is moot.
The first BXD mappings above passed all 235 columns with null for the
F1s (and for every strain the trait does not cover), i.e. the phenotype
set was the trait's 34 strains while the genotype side spanned 235. That
is not GN2's default, and it changes GEMMA's kinship matrix (computed over
genotypes, not phenotypes) and R/qtl's cross. Measured on trait 10001
both ways, it happened not to change the answer:
| columns passed | GEMMA best | R/qtl best LOD |
|---|---|---|
| all 235 (strains + F1s) | p 5.9e-07 at rsm10000005700, chr 8 |
2.9320 at rs48756159, chr 8 |
| 198 strains only | p 5.9e-07 at rsm10000005700, chr 8 |
2.9320 at rs48756159, chr 8 |
So null-filling is enough to keep the phenotypes right, but not to
exclude an individual from the model: the genotype matrix has to be
subset. When the API takes over the alignment, it should subset to the
strains by default and offer "include parents and F1s" explicitly, which
is what GN2's checkbox does.
No restart is needed to use a new genotype file
The path travels in each request and nothing in the server caches a
panel, so pointing a request at a newer file (BXD.8.geno instead of
BXD.geno, HET3-ITP.12.geno instead of .11) takes effect on the next
request. Both engines' result caches key on file contents, not paths:
src/rqtl.rs reads the genotype file and hashes it into the output
filename, and gemma-wrapper names its cached kinship by a content hash --
so a file that has changed is recomputed, not silently reused.
What does need a restart:
- a new binary (any Rust change), and
GENENETWORK4_TEST_TOKEN, which is decided when the router is built; - the process environment:
PATHandR_LIBS_SITE(R/qtl'sRscriptand itsqtl/optparse/stringi/stringrpackages), and the values the process fixes at first use --SQL_URI(the MySQL pool),XAPIAN_DB_PATH/SEARCH_ENGINE(the open index handle); - not
scripts/rqtl_wrapper.R: it runs as a subprocess per request, so editing it takes effect immediately.
What is still messy
HET3-ITP has eleven .geno versions on disk (.2 … .12) and no
genotype/HET3-ITP.json, so there is no recorded list and no recorded
"current" — .12 (6447 individuals, 894 markers, Apr 2023) is the newest by both
numbering and date -- the rqtl2/ bundles, for what it is worth, only
exist for .2, .3 and .4, so they do not mark the current version
either -- but that is inference, not a documented default. Its traits
are per animal, not per strain (lifespan: 23 879 values over
several cohorts), so only 6 438 of them fit the .12 panel; the rest
belong to cohorts that genotype file does not contain. Mapping them
"latest version" therefore silently drops cohorts — correct per the
file, wrong as biology until the version/cohort pairing is recorded.
The fix on the GN4 side is small and worth doing: align trait values
onto the panel inside the compute handlers (by name, from the .geno
header), so a caller passes the values it has and the server does the
re-indexing. That also makes the actions bodies directly usable.
