diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-07 16:28:14 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-07 16:28:14 +0000 |
| commit | 14dc58f070c2950c13ae822c4a371cf3d303b16a (patch) | |
| tree | 60799fc8b97344f496205dd3fea59edff0d307d1 /docs | |
| parent | b5f5209642f422e87a4d0041471c5654491a8d0c (diff) | |
| download | genenetwork4-14dc58f070c2950c13ae822c4a371cf3d303b16a.tar.gz | |
docs: name the real genotype files, panels and timings everywhere
docs/genotypes.md explained the convention; the endpoint docs still left a caller guessing which file to pass and how big the panel is. So: * README.md (/compute/gemma): GEMMA wants the BIMBAM pair `genotype/bimbam/<file>_geno.txt` + `<file>_snps.txt`, not the `.geno` file; `trait_values` is one entry per individual in that file's column order; BXD's current panel stated (235 x 21056), with the 17.1 s measurement against the 19 s the 7 320-marker fixtures take. * README.md (/compute/rqtl): the file is `genotype/<file>.geno` (BXD.8: 235 x 21057; HET3-ITP.12: 6447 x 894), `MappingMethodId` decides the engine, and the two measured runs with the UM-HET3 peak. * README.md (/compute/gemma/ravanan): same input convention as gemma. * docs/genotypes.md: how each measurement was made, step by step, so they can be repeated; and that step 3 (expanding values onto the panel) is the API's job in the next iteration. * docs/www/ai-instruction.md (4a): a table of the BXD files per engine with panel sizes, the UM-HET3 file, "ask the operator for any other panel" with where GN2 records which one is current, and the measured times. Values must be one per individual, in file order -- position, not name -- and the API does not re-index yet, so a mapping is prepared for the caller rather than guessed. * docs/www/operator-checklist.md: MappingMethodId per group, and the three measurements.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/genotypes.md | 19 | ||||
| -rw-r--r-- | docs/www/ai-instruction.md | 39 | ||||
| -rw-r--r-- | docs/www/operator-checklist.md | 12 |
3 files changed, 61 insertions, 9 deletions
diff --git a/docs/genotypes.md b/docs/genotypes.md index fed2932..e2d91da 100644 --- a/docs/genotypes.md +++ b/docs/genotypes.md @@ -99,6 +99,25 @@ and were posted to the API. | BXD, `BXD.8.geno` | R/qtl scanone | 10001 | 235 | 34 | 7.5 s | 21 056 markers | | HET3-ITP, `HET3-ITP.12.geno` (Apr 2023) | R/qtl scanone | 10000 lifespan | 6 447 | 6 438 | 83.1 s | 893 markers, peak LOD 7.95 at `1_24042124` (chr 1, 24.04 Mb) | +## Reproducing the measurements + +The pattern for all three, and for any mapping this deployment runs: + +1. take the sample order from the genotype file's header (first non-`#`/ + `@` line, column 5 on if column 4 is `Mb`, else column 4 on); +2. fetch `/groups/{group}/traits/{id}/values` and index those values by + sample name; +3. build one entry per panel member, in header order, `null` where the + trait has none; +4. POST that to `/compute/rqtl` (`samples: [{name, value}]` with + `genotype_file: .../<file>.geno`) or `/compute/gemma` + (`trait_values: [...]` with the `bimbam/<file>_geno.txt` and + `<file>_snps.txt` pair). + +Step 3 is the API's job in the next iteration; until then, a request +built without it is caught rather than trusted -- R/qtl refuses a count +mismatch, and GEMMA fails on one. + ## What is still messy `HET3-ITP` has eleven `.geno` versions on disk (`.2` … `.12`) and **no diff --git a/docs/www/ai-instruction.md b/docs/www/ai-instruction.md index 0c8bfb4..e6bdb8c 100644 --- a/docs/www/ai-instruction.md +++ b/docs/www/ai-instruction.md @@ -262,18 +262,39 @@ itself, and the choice of genotypes, is the server's. the server's own filesystem. A valid token is the only requirement: they are not dataset-checked, so nothing about §4 applies — but the paths you name have to be paths the server can read, which means ones the -deployment already holds. You cannot upload a genotype file, and you -should not go looking for one: ask the operator which paths the -deployment uses for the panel you are working on. +deployment already holds. You cannot upload a genotype file. + +Which file, and which panel version, is a deployment convention you +cannot guess; the current BXD panel is + +| engine | file | panel | +|---|---|---| +| R/qtl | `genotype/BXD.8.geno` (WGS-based, Mar 2022) | 235 individuals × 21 057 markers | +| GEMMA | `genotype/bimbam/BXD.8_geno.txt` + `BXD.8_snps.txt` | 235 individuals × 21 056 markers | + +under `/home/wrk/tmp/ai-root/factory/gn4-data/genotype-files/`, alongside +`genotype/HET3-ITP.12.geno` (6 447 individuals × 894 markers; UM-HET3 is +a 4-way cross, so R/qtl is the engine there, not GEMMA). Ask the operator +for the path and version for any *other* panel: which file is current is +recorded in GeneNetwork2's per-group `genotype/<Group>.json`, whose +`genofile` list is newest-first, and `InbredSet.MappingMethodId` says +which engines a group offers at all. The values and the genotype file have to describe the **same -individuals, in the same order, one for one**. Both engines match them by +individuals, in the same order, one for one**: one entry per panel +member, `null` where the trait has no value. Both engines match by position, not by name. Get that wrong and the computation is meaningless; -GEMMA refuses outright, and R/qtl now refuses too rather than returning -plausible-looking numbers for the wrong strains. So: use the values from -the values endpoint and the genotype file the deployment pairs with that -panel — never a genotype file from a different panel, and never a -hand-built trait vector. +GEMMA refuses outright, and R/qtl refuses a count mismatch rather than +returning plausible-looking numbers for the wrong strains. + +The API does not re-index a trait's values onto the panel yet, so a +mapping request is something the operator prepares: the panel's sample +order lives in the genotype file's header, not in this API. Ask for the +mapping (or for the panel order) rather than guessing — and never use a +genotype file from a different panel, or a hand-built trait vector. For +reference, the deployment maps BXD (235 individuals, 34 with values) in +about 17 s with GEMMA and 8 s with R/qtl, and UM-HET3 lifespan (6 447 +individuals, 893 markers) in about 83 s with R/qtl. Results are **cached** by input, so running the same request twice costs little the second time (a cached GEMMA run on the BXD fixtures comes back diff --git a/docs/www/operator-checklist.md b/docs/www/operator-checklist.md index 333ed68..f21337f 100644 --- a/docs/www/operator-checklist.md +++ b/docs/www/operator-checklist.md @@ -147,6 +147,18 @@ re-indexing yet, so whoever prepares the request has to. It is the first thing to fix: pass the values a caller has, let the server align them from the `.geno` header. +Which engines a group offers is `InbredSet.MappingMethodId` (1 → GEMMA, +QTLReaper, R/qtl; 2 → GEMMA; 3 → R/qtl; 4 → GEMMA, PLINK): BXD = 1, +HET3-ITP = 3. So UM-HET3 is R/qtl only, correct for a 4-way cross even +though `bimbam/HET3-ITP.*` exists. + +Measured on this box with the newest genotypes: BXD (`BXD.8`, 235 +individuals, trait 10001 with 34 values) 17.1 s / 21 055 marker rows on +GEMMA and 7.5 s / 21 056 markers on R/qtl; UM-HET3 lifespan over +`HET3-ITP.12` (6 447 individuals, 6 438 with values) 83.1 s / 893 +markers, peak LOD 7.95 at chr 1, 24.04 Mb. Details and the misgivings +about HET3-ITP's versioning are in `docs/genotypes.md`. + `/compute/gemma/ravanan` additionally needs `ravanan` and `ccwl` on `PATH`, and with `RAVANAN_BATCH_SYSTEM=slurm-api` a reachable slurmrestd (see `docs/ravanan-slurm.md`). |
