about summary refs log tree commit diff
path: root/scripts/gemma-compare.py
AgeCommit message (Collapse)Author
47 hoursgemma-compare: make the parity gate run, and say what it is measuringPjotr Prins
The script could not have caught anything: it posted without an Authorization header (401 against a token-protected server), asked for a trait's values as `{"name": n, "values": [<trait name>]}` (422: samples need numbers), read `columns`/`rows` from the envelope's top level (they are under `meta` and `data`), and compared relatively -- which calls beta 0.0131 against 0.0157 a 16% disagreement on a marker where both runs agree to three decimals. Now: `--token`/`$GN4_TOKEN`, trait values fetched from the API, the envelope shape handled, and an ABSOLUTE `--tolerance` (default 0 = exact). It fetches, compares and reports through the server as deployed. Verdict on the current deployment, BXDPublish 10001 over 34 individuals: rows: gemma-wrapper 21055, ravanan+gemma 21055, shared 21055 worst absolute difference: rs230640887 beta -1.015599 vs -1.018739 (0.00314) --tolerance 0.005 -> agree; --tolerance 0 -> 63 163 values beyond it So the row difference this started from is gone: same markers, same count. The residual is that the two paths run *different GEMMA builds* -- the wrapper's log names gemma-gn2-0.98.5-8cd4cdb, the ravanan manifest gemma 0.98.6 -- which moves beta/se/p_lrt in the third decimal while leaving af, n_miss, l_mle and the marker set identical (and the top hits in the same order).
4 daysravanan+gemma: call gemma itself, not gemma-wrapperPjotr Prins
gemma-wrapper was doing two things ravanan now does or GEMMA already does: the parallel fan-out (ravanan's CWL scatter replaced it long ago, which is why every job passed --no-parallel) and LOCO orchestration -- gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and takes the annotation file it needs. What was left was a content-hash K cache and a --json record with results packed into .tar.xz archives. So cwl/gemma/gemma-loco.scm now runs two gemma invocations per chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix> -lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat, deterministic names. The kinship matrix travels from one step to the other as a CWL File output, and each job's .assoc.txt and .log.txt are the workflow's outputs -- so compute_ravanan reads files instead of parsing a wrapper JSON record and unpacking archives, and the pve note comes straight out of the jobs' logs. Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows -- 636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper also needed Ruby and pfff on PATH, and named its outputs with the extension doubled (...1.assoc.txt.assoc.txt). /compute/gemma keeps gemma-wrapper, deliberately: that is what makes the two endpoints comparable rather than two copies of one stack. scripts/gemma-compare.py runs the same body through both and reports row-level differences and timings. The gemma-wrapper content-hash K cache is the one thing given up; a repeat request recomputes (271 ms of kinship on this fixture).