about summary refs log tree commit diff
path: root/scripts
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-10 21:34:36 +0000
committerPjotr <wrk@bosa.lan>2026-11-10 21:34:36 +0000
commit92a7d0282897dd056b5694a20b2aee7cdc1ce6e3 (patch)
tree72819710f2032da3301c6f3f3f24e7753cf074d2 /scripts
parent1ee249766366f3d88c1486065f38a0fc58b8171b (diff)
downloadgenenetwork4-92a7d0282897dd056b5694a20b2aee7cdc1ce6e3.tar.gz
uploads: a genotype matrix that updates an existing population
Tab-delimited, header names the individuals from column 4, then marker,
chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own
SQLite file -- nothing goes near MariaDB -- and written out on demand as
a GN .geno carrying the base's @-metadata, so the R scripts read exactly
what was uploaded.

An upload updates a population, it does not invent one, and the checks
say so:

  * the base version must exist (resolved through the same
    genotypes::resolve_named the rest of the API uses);
  * at most 50 individuals the base does not have, and at least one that
    it does -- no overlap is a different population, not an update;
  * at most 200 000 markers, and the refusal says the sysadmin has to
    organize a file that size, because this is a request path, not an
    ingest;
  * the values must be the same kind as the base's: letters, whole
    numbers or probabilities. A letters panel holding 0/1 would be read
    as the wrong thing silently, so it is refused naming both kinds.

GET /uploads/genotypes lists them, /{name} its metadata (overlap, new
individuals, kind, digest), /{name}/markers pages the marker table, and
/{name}/geno writes the file and returns the path for `genotype_file` on
the compute endpoints. DELETE removes one.

scripts/example-genotype-upload.py turns any .geno into a valid update --
a subset of markers, a few individuals dropped and added -- and
tests/genoupload.rs builds its own from the deployment's BXD.4 for the
same reason: upload, page the markers, write the .geno and check the
layout, delete, then break each rule (no overlap, numbers in a letters
panel, commas) and check the refusal says which.

Unit tests cover the three value kinds, missing cells, ragged rows, a
non-numeric position, the 50-new-individual limit, and the .geno layout
(Chr first, then the marker -- GN's order, not the upload's).
Diffstat (limited to 'scripts')
-rwxr-xr-xscripts/example-genotype-upload.py71
1 files changed, 71 insertions, 0 deletions
diff --git a/scripts/example-genotype-upload.py b/scripts/example-genotype-upload.py
new file mode 100755
index 0000000..0e36e63
--- /dev/null
+++ b/scripts/example-genotype-upload.py
@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+"""Write an example genotype *upload* from an existing .geno file.
+
+The upload format GN4 accepts is tab-delimited: the header names the
+individuals from column 4 on, column 1 is the marker, 2 the chromosome, 3 the
+position in Mb. This script converts a GN `.geno` into that shape, keeping a
+subset of the markers, dropping a few individuals and (optionally) adding new
+ones -- which is exactly what an "update an existing population" upload is.
+
+    scripts/example-genotype-upload.py \\
+        --geno /home/wrk/tmp/ai-root/factory/gn4-data/genotype-files/genotype/BXD.4.geno \\
+        --markers 20 --drop 2 --add 1 --out /tmp/example-upload.tsv
+"""
+
+import argparse
+import sys
+
+
+def read_geno(path: str):
+    """(individuals, markers) from a GN .geno file."""
+    labels, individuals, markers = [], [], []
+    with open(path, errors="replace") as handle:
+        for line in handle:
+            line = line.rstrip("\n")
+            if not line.strip():
+                continue
+            if line.startswith("#"):
+                continue
+            if line.startswith("@"):
+                labels.append(line)
+                continue
+            fields = line.split("\t")
+            if not individuals:
+                start = 4 if len(fields) > 3 and fields[3] == "Mb" else 3
+                individuals = [name.strip() for name in fields[start:]]
+                continue
+            start = 4 if len(fields) > 4 else 3
+            markers.append((fields[1].strip(), fields[0].strip(), fields[start - 1].strip(),
+                            [cell.strip() for cell in fields[start:]]))
+    return individuals, markers
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--geno", required=True, help="a GN .geno file to convert")
+    parser.add_argument("--out", required=True, help="where to write the upload")
+    parser.add_argument("--markers", type=int, default=20, help="how many marker rows to keep")
+    parser.add_argument("--drop", type=int, default=0, help="individuals to leave out")
+    parser.add_argument("--add", type=int, default=0, help="new individuals to add (max 50)")
+    arguments = parser.parse_args()
+
+    individuals, markers = read_geno(arguments.geno)
+    if not individuals or not markers:
+        raise SystemExit(f"{arguments.geno}: no individuals or markers")
+    kept = individuals[: len(individuals) - arguments.drop] if arguments.drop else individuals
+    added = [f"NEW{i + 1}" for i in range(arguments.add)]
+    header = ["marker", "chr", "pos", *kept, *added]
+    with open(arguments.out, "w") as handle:
+        handle.write("\t".join(header) + "\n")
+        for name, chr, pos, cells in markers[: arguments.markers]:
+            # A new individual inherits the first kept individual's calls: a
+            # valid value of the same kind, which is all the upload checks.
+            row = [name, chr, pos, *cells[: len(kept)], *([cells[0]] * len(added))]
+            handle.write("\t".join(row) + "\n")
+    print(f"{arguments.out}: {len(kept)} kept + {len(added)} new individual(s), "
+          f"{min(arguments.markers, len(markers))} marker(s)", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())