about summary refs log tree commit diff
path: root/cwl/README.md
AgeCommit message (Collapse)Author
5 daysravanan: key the kinship step on the animal set, so K is shared across traitsPjotr Prins
gemma -gk reads the phenotype file only to learn which individuals are analysed -- the matrix comes from the genotypes -- so the workflow now gives the kinship step a constant-valued pheno file carrying just the trait's missing-data pattern, and gives gwa the real values. Verified byte-for-byte: -gk on a real trait and on its constant-valued twin produce identical .cXX.txt files, and trait 2's assoc checksums are identical whether K came from the constant pheno or the real one. Consequence, measured on the BXD fixture: a second trait over the same animals costs +1.0 MB of store and 7.5 s (20 GWA jobs) instead of +13 MB and 14.6 s (a second kinship scatter), because the 20 K matrices are identical inputs for both traits. Three keys now: workflow (the ccwl source), kinship (genotype + snps + lmm + the NA pattern) and trait (kinship key + the values), so the compiled CWL and the kinship pheno file are shared and only the trait's pheno file and inputs YAML are per-trait. This is also stricter than gemma-wrapper's own key, which ignores the phenotype entirely and would reuse a K computed for a different set of animals.
5 dayscwl: record the cold and warm LOCO timings, Slurm includedPjotr Prins
Full 20-chromosome LOCO mapping on the BXD fixture: 14.3 s cold and 1.6 s warm single-machine, 275.6 s cold and 1.6 s warm through the Slurm test container -- 40 jobs, 7320 rows, identical .assoc.txt checksums in every case. Slurm costs ~6.9 s per job here against ~0.36 s single-machine, which is container and scheduling overhead rather than gemma: the first request is cheaper locally, and only a repeated request (answered from ravanan's store) is fast either way.
5 daysravanan: name the workflow inputs by content, so a repeat request is warmPjotr Prins
compute_ravanan wrote <pid>-<nanos>-pheno.txt, -ravanan-inputs.yaml and -gemma-loco.cwl, so ravanan's (script, inputs) work-reuse key was new on every call and a repeat of the identical request recomputed all 40 steps. Naming them by a hash of the request's inputs (genotype path and size, snps contents, trait values, lmm) makes an identical request hit the store instead. Measured on the BXD fixture, full 20-chromosome LOCO mapping, single-machine: 14.3 s cold, 1.6 s warm, identical .assoc.txt checksums. For comparison the gemma-wrapper stack costs 11.9 s cold and 12.1 s warm -- its own cache did not hit on a repeat here -- so the ravanan path is now the cheaper one for any repeated request and the slightly dearer one for a first request. The run log stays per-call: it is what a caller streams by run_id. The three shared files are written aside and renamed, so two concurrent identical requests cannot read a half-written file.
5 daysravanan+gemma: call gemma itself, not gemma-wrapperPjotr Prins
gemma-wrapper was doing two things ravanan now does or GEMMA already does: the parallel fan-out (ravanan's CWL scatter replaced it long ago, which is why every job passed --no-parallel) and LOCO orchestration -- gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and takes the annotation file it needs. What was left was a content-hash K cache and a --json record with results packed into .tar.xz archives. So cwl/gemma/gemma-loco.scm now runs two gemma invocations per chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix> -lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat, deterministic names. The kinship matrix travels from one step to the other as a CWL File output, and each job's .assoc.txt and .log.txt are the workflow's outputs -- so compute_ravanan reads files instead of parsing a wrapper JSON record and unpacking archives, and the pve note comes straight out of the jobs' logs. Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows -- 636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper also needed Ruby and pfff on PATH, and named its outputs with the extension doubled (...1.assoc.txt.assoc.txt). /compute/gemma keeps gemma-wrapper, deliberately: that is what makes the two endpoints comparable rather than two copies of one stack. scripts/gemma-compare.py runs the same body through both and reports row-level differences and timings. The gemma-wrapper content-hash K cache is the one thing given up; a repeat request recomputes (271 ms of kinship on this fixture).
14 daysgemma: key the gemma-wrapper cache dir by input setPjotr
The cache dir was keyed by chromosome only and shared across requests, so a request with different inputs (another trait, or another snps or genotype file) left a second *-gemma-cXX.tar.xz in the same chr-N dir. gemma-wrapper names those archives by a content hash, so the workflow's single-File glob output binding then matched more than one path and the step failed -- silently, since ravanan's user-error only logs when %log-level is set, which never happens inside a generated job script. Key the cache root by a hash of the input set (genotype path and size, snps contents, trait values, lmm) as well as by chromosome, so repeated identical requests still hit gemma-wrapper's cache while different input sets cannot see each other's archives. chromosomes_in reads the snps file, so it now takes the already-read contents (chromosomes_from) to avoid reading it twice. cwl/README.md: describe the two-part keying, and note that a node needs a writable HOME -- ravanan forwards the submitting process's HOME into the job (as sbatch would) and gemma-wrapper writes its lock file there.
2026-10-24cwl: Expand the "rebuild on the nodes instead of sharing a store" section.Pjotr
Spell out why the current design needs the store visible (ravanan submits a store path and Slurm does not substitute, so an absent path is an immediate ENOENT), then list the five ways out as a table -- shared store, realise-then-run wrapper, per-workflow realise job, guix publish plus wrapper, shipping closures -- with what runs, what each needs on the nodes, and what it costs in caching. Also records where the wrapper would live (build-gexp-script already builds the derivation but returns only its output path; submit-job in ravanan/slurm-api.scm assembles the script field), that nothing of it is implemented here, and the four costs that decide the question: channel skew, build time inside the job's time limit, caching on ephemeral node stores, and guix publish as the sane companion. Tables tightened so no line runs past ~170 characters.
2026-10-24cwl: Document how a ravanan+Guix+Slurm request actually travels.Pjotr
cwl/README.md explains the whole path (API -> ccwl compile -> ravanan -> slurmrestd -> slurmd), where Guix sits (front end only: nodes just need the same /gnu/store mounted, plus bash, a writable ravanan store for data/logs, and scratch for the job's working directory), the two caches (ravanan's work reuse, checked before submitting; gemma-wrapper's per-chromosome --cache-dir, inside the job) and why the latter is what actually saves work for this endpoint. It also answers "can the nodes rebuild the tools instead of sharing a store": yes in principle, since Guix is deterministic and a node building the same derivations would materialise identical store paths, but ravanan submits a store path and Slurm does not substitute automatically -- so it needs a realise-then-run wrapper (ravanan's slurm backend already holds the script's derivation), a per-workflow realise job, or guix publish plus a node-side daemon, with the costs and channel-pinning caveats listed. The timing table from the test cluster (cached: identical; fresh: waves vs synchronous steps) is included so expectations are explicit. docs/ravanan-slurm.md and README.md link to it.