about summary refs log tree commit diff

Genotype files: where they are, which one is current, how to map with them

/compute/rqtl and /compute/gemma take paths that exist on the server, so whoever calls them has to know two things the API does not tell them: where the deployment keeps genotype files, and which version of a panel's genotypes is the current one. Both are answered by long-standing GeneNetwork convention, reproduced here because it is not discoverable from this repository's code.

The deployment's copy is /home/wrk/tmp/ai-root/factory/gn4-data/ genotype-files, laid out like GeneNetwork2's own genotype_files:

genotype/<Group>.geno                 GN genotype format (Chr/Locus/cM/Mb + one column per individual)
genotype/<Group>.json                 per-group metadata: the list of genotype files, newest first
genotype/bimbam/<file>_geno.txt       GEMMA/BIMBAM genotype (0/0.5/1) + <file>_snps.txt + <file>_pheno.txt
genotype/rqtl2/<file>_{geno,gmap,pheno}.csv, <file>.json    R/qtl2 bundle
genotype/json/<Group>.json            older marker dump (not the version list)
mapping/<Group>.bed|.bim|.fam         PLINK, with covariates in mapping/

Which file is current

GeneNetwork2 (gn2/base/data_set/datasetgroup.py:get_genofiles, gn2/base/webqtlConfig.py:GENODIR): read genotype/<Group>.json and take its genofile key, which is a list, newest first, each entry {title, location, sample_list?}. The mapping UI offers the list and defaults to the first. With no JSON, it falls back to <Group>.geno. The user's choice rides along in genofile_string as <file>:<title> (display_mapping_results.py).

So "the latest version" is genofile[0], not the highest number and not the newest mtime. For BXD that is:

{"title": "WGS-based (Mar2022)", "location": "BXD.8.geno", "sample_list": [...235 names...]}

followed by BXD.geno ("Array-based 2017–2021"), BXD.4.geno ("Classic 2001-2016") and BXD.2.geno ("Test"). Note BXD.8.geno is not even in the GenoFile table — that table is older and only ever had the classic/array files — so the JSON, not MariaDB, is the authority on which version is current. GenoFreeze is one row per group ("BXD Genotypes", 2003), not one per version.

How each engine is fed

  • GEMMA (GN2 marker_regression/gemma_mapping.py): strip .geno from the location and use genotype/bimbam/<name>_geno.txt with <name>_snps.txt — for BXD's current file that is bimbam/BXD.8_geno.txt + bimbam/BXD.8_snps.txt (21 056 markers × 235 individuals). The .geno file itself is not what GEMMA gets.
  • R/qtl (GN2 marker_regression/rqtl_mapping.py): pass genotype/<location> straight through — genotype/BXD.8.geno — and let scripts/rqtl_wrapper.R's geno_to_csvr read the GN format.
  • The sample list (GN3 gn3/db/genotypes.py:load_genotype_samples, gn3/db/datasets.py): the first non-#/@ line of the .geno file is the header; if its fourth column is Mb, sample names start at column 5, otherwise at column 4. That is the only authoritative sample order, and the only place it is stored.

The alignment rule, which the API now applies

Both engines match values to individuals by position: R/qtl's cross format has no per-sample names, and write_pheno_file (src/gemma.rs) writes bare values. Inside the API that no longer leaks out: a request names a group (and optionally a version), passes the trait's values as named samples, and src/genotypes.rs aligns them onto the panel by name, dropping names the panel does not have and filling null for individuals the trait does not cover. Parents and F1s are excluded unless include_parents_f1 is set — and excluded the only way that works, by subsetting the genotype matrix (a cached panel-subset file under the engine's cache directory), because null-filling keeps an individual out of the fit but not out of GEMMA's kinship matrix or R/qtl's cross.

scripts/rqtl_wrapper.R still refuses a count mismatch, which is what catches a hand-built request that bypasses the alignment (an explicit genotype_file with the wrong number of samples).

Which engine a group offers

InbredSet.MappingMethodId (GN2 get_mapping_methods): 1 → GEMMA, QTLReaper and R/qtl; 2 → GEMMA; 3 → R/qtl; 4 → GEMMA and PLINK. On this database: BXD = 1, HET3-ITP = 3, HSNIH-Palmer = 2, HS = NULL.

HET3-ITP is UM-HET3, a 4-way cross (@type:4-way, @mat:AB, @pat:CD in its .geno header), so R/qtl is the right engine even though bimbam/HET3-ITP.* files exist — those are a biallelic projection of a four-founder panel.

Measured runs (2026-11-07, this deployment)

Values came from /groups/{group}/traits/{id}/values, were re-indexed onto the genotype file's sample list by name with null for the rest, and were posted to the API.

panel / genotype file engine trait panel trait values time result
BXD, BXD.8.geno (WGS Mar2022) GEMMA (bimbam) 10001 cerebellum weight 235 34 17.1 s 21 055 marker rows
BXD, BXD.8.geno R/qtl scanone 10001 235 34 7.5 s 21 056 markers
HET3-ITP, HET3-ITP.12.geno (Apr 2023) R/qtl scanone 10000 lifespan 6 447 6 438 83.1 s 893 markers, peak LOD 7.95 at 1_24042124 (chr 1, 24.04 Mb)

What the API tells a caller about a genotype file

Three routes, all behind the bearer token like every other endpoint:

route what it gives
GET /groups/{group}/genotypes versions newest-first, each with its file name, title, individuals, markers, which engines can use it, and which is current
GET /groups/{group}/genotypes/{version} that file's file name, its header @-labels, the panel's individuals in file order, the strains a mapping uses, counts, and per-chromosome marker counts
GET /groups/{group}/genotypes/{version}/markers?page=&limit=&chr= the marker table: name, chromosome, cM, Mb — paged, positional rows

The genotype calls are never served. A caller sees the panel, the markers and the header, which is what it needs to frame a mapping and to check one; the matrix itself stays on the server, and the compute endpoints take a group and a version rather than a path.

Reproducing the measurements

The measurements above were taken before the API took over the resolution, by doing its steps by hand. Now it is two calls:

# what can BXD be mapped with, and which version is current?
curl -s -H "Authorization: Bearer $TOKEN" "$API/groups/BXD/genotypes"

# map a trait over it: group + the trait's own values, no paths
curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -X POST "$API/compute/rqtl" -d '{
    "group": "BXD",
    "trait_name": "BXDPublish::10001",
    "samples": [{"name": "BXD1", "value": 61.4}, ...]
  }'

Add "genotype_version": "BXD.4" to map an older panel. The response's notes names the resolved version, the panel size after excluding parents/F1s, and how many supplied values aligned — that line is what makes a run reproducible without publishing paths.

Parents and F1s are excluded by default

GeneNetwork2 and GeneNetwork3 map without the parents and F1s unless asked otherwise: GN2's DatasetGroup.incparentsf1 defaults to False (gn2/base/data_set/datasetgroup.py) and its ParInfo parents are only added to the genotype object when it is on; GN3 spells the same thing as get_sample_list(group, inc_par=False, inc_f1=False) (gn3/db/datasets.py). The parent/F1 lists themselves come from parents_and_f1s.json in the data directory.

On this deployment the distinction is narrower than it sounds, because the panels already tell you what they hold:

  • BXD.8.geno is 235 columns = 198 RI strains + 37 F1s, and no parent columns at all (C57BL/6J and DBA/2J are not in the file; the JSON's 235-entry sample_list is the same 198+37). So "exclude parents and F1s" here means "use the 198 strains".
  • HET3-ITP.12.geno is 6 447 individual animals (1 953 JL… plus 4 494 UM…), no parents, no F1s — UM-HET3 maps individuals, so the default is moot.

The first BXD mappings above passed all 235 columns with null for the F1s (and for every strain the trait does not cover), i.e. the phenotype set was the trait's 34 strains while the genotype side spanned 235. That is not GN2's default, and it changes GEMMA's kinship matrix (computed over genotypes, not phenotypes) and R/qtl's cross. Measured on trait 10001 both ways, it happened not to change the answer:

columns passed GEMMA best R/qtl best LOD
all 235 (strains + F1s) p 5.9e-07 at rsm10000005700, chr 8 2.9320 at rs48756159, chr 8
198 strains only p 5.9e-07 at rsm10000005700, chr 8 2.9320 at rs48756159, chr 8

So null-filling is enough to keep the phenotypes right, but not to exclude an individual from the model: the genotype matrix has to be subset. When the API takes over the alignment, it should subset to the strains by default and offer "include parents and F1s" explicitly, which is what GN2's checkbox does.

No restart is needed to use a new genotype file

The path travels in each request and nothing in the server caches a panel, so pointing a request at a newer file (BXD.8.geno instead of BXD.geno, HET3-ITP.12.geno instead of .11) takes effect on the next request. Both engines' result caches key on file contents, not paths: src/rqtl.rs reads the genotype file and hashes it into the output filename, and gemma-wrapper names its cached kinship by a content hash -- so a file that has changed is recomputed, not silently reused.

What does need a restart:

  • a new binary (any Rust change), and GENENETWORK4_TEST_TOKEN, which is decided when the router is built;
  • the process environment: PATH and R_LIBS_SITE (R/qtl's Rscript and its qtl/optparse/stringi/stringr packages), and the values the process fixes at first use -- SQL_URI (the MySQL pool), XAPIAN_DB_PATH/SEARCH_ENGINE (the open index handle);
  • not scripts/rqtl_wrapper.R: it runs as a subprocess per request, so editing it takes effect immediately.

What is still messy

HET3-ITP has eleven .geno versions on disk (.2 … .12) and no genotype/HET3-ITP.json, so there is no recorded list and no recorded "current" — .12 (6447 individuals, 894 markers, Apr 2023) is the newest by both numbering and date -- the rqtl2/ bundles, for what it is worth, only exist for .2, .3 and .4, so they do not mark the current version either -- but that is inference, not a documented default. Its traits are per animal, not per strain (lifespan: 23 879 values over several cohorts), so only 6 438 of them fit the .12 panel; the rest belong to cohorts that genotype file does not contain. Mapping them "latest version" therefore silently drops cohorts — correct per the file, wrong as biology until the version/cohort pairing is recorded.

The fix on the GN4 side is small and worth doing: align trait values onto the panel inside the compute handlers (by name, from the .geno header), so a caller passes the values it has and the server does the re-indexing. That also makes the actions bodies directly usable.