about summary refs log tree commit diff
path: root/README.md
diff options
context:
space:
mode:
authorPjotr Prins <pjotr.public01@thebird.nl>2026-10-19 07:13:15 +0000
committerPjotr Prins <pjotr.public01@thebird.nl>2026-10-19 07:13:15 +0000
commitd504b5da9f5deeece9ba2ff4dcd8baf3dfa4dc32 (patch)
treeeaed193876da61ad7d6ae55906dccf4c6198c583 /README.md
parent477294a2aa9ae7348137271b66831ac93a6149d6 (diff)
downloadgenenetwork4-gemma-cwl-parallel.tar.gz
gemma: compile ravanan's ccwl workflow on demand, don't check in the CWL gemma-cwl-parallel
gemma-loco.cwl was a generated artifact (ccwl compile's output), but
was being committed like hand-written source. Remove it, and have
compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL
file under its own cache dir on every call instead -- cheap (pure
macro expansion, no Guix build involved), so there's no real cost to
regenerating it fresh each time, and no risk of a stale compiled file
drifting from its ccwl source.

Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH,
alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now
names the .scm source, not a .cwl file. Verified end to end by running
the exact compile-then-run sequence compute_ravanan now performs
against the real ccwl/ravanan/gemma-wrapper binaries and the BXD
fixtures: same correctness and timings as before.
Diffstat (limited to 'README.md')
-rw-r--r--README.md37
1 files changed, 21 insertions, 16 deletions
diff --git a/README.md b/README.md
index 5c5719f..421e56f 100644
--- a/README.md
+++ b/README.md
@@ -1002,32 +1002,37 @@ yet) is a 404, not an empty response -- genuinely different from
 
 Same request/response shape as `/compute/gemma`, same two gemma-wrapper
 steps (kinship, then GWA, both `--loco`), but driven by
-[ravanan](https://github.com/vepiteski/ravanan) running the CWL
-workflow at `cwl/gemma/gemma-loco.cwl` instead of this server's own
+[ravanan](https://github.com/vepiteski/ravanan) running a CWL workflow
+compiled on every call (via `ccwl compile`, see below) from the ccwl
+source at `cwl/gemma/gemma-loco.scm`, instead of this server's own
 direct subprocess calls -- an alternative runner for the same
 computation, kept side by side with `/compute/gemma` so the two can be
 compared (correctness, timing). Unlike gemma-wrapper's own `--loco`
 mode, which fans its per-chromosome kinship/GWA jobs out internally via
 GNU `parallel`, this workflow has ravanan itself do that fan-out: it
 scatters one `gemma-wrapper --chromosomes <chr> --no-parallel ...`
-invocation per chromosome (written in
+invocation per chromosome, each restricted to exactly the chromosome
+it was given (gemma-wrapper itself still does the correct "exclude
+this chromosome from kinship, test only its markers" LOCO logic,
+against the *full* genotype/SNP files -- `--chromosomes` only
+overrides which chromosomes gemma-wrapper loops over, not the
+per-chromosome math itself). This workflow is written in
 [ccwl](https://ccwl.systemreboot.net/) at `cwl/gemma/gemma-loco.scm`,
-compiled to `gemma-loco.cwl` with `ccwl compile`), each restricted to
-exactly the chromosome it was
-given (gemma-wrapper itself still does the correct "exclude this
-chromosome from kinship, test only its markers" LOCO logic, against
-the *full* genotype/SNP files -- `--chromosomes` only overrides which
-chromosomes gemma-wrapper loops over, not the per-chromosome math
-itself). `compute_ravanan` computes the chromosome list from
-`snps_file` itself (gemma-wrapper's own `get_chromosomes`, reproduced
-as `chromosomes_in`) and passes it as the workflow's `chromosomes`
-input, since CWL's `scatter` needs a concrete array at invocation
-time. `ravanan_cmd`, `gemma_ravanan_workflow`, `gemma_ravanan_manifest`
+not raw CWL; `compute_ravanan` compiles it to CWL with `ccwl compile`
+fresh on every call (cheap -- pure macro expansion, no Guix build
+involved) into a generated, throwaway file under `ravanan_cache_dir`,
+never checked into the repo. `compute_ravanan` computes the chromosome
+list from `snps_file` itself (gemma-wrapper's own `get_chromosomes`,
+reproduced as `chromosomes_in`) and passes it as the workflow's
+`chromosomes` input, since CWL's `scatter` needs a concrete array at
+invocation time. `ravanan_cmd`, `ccwl_cmd`, `gemma_ravanan_workflow`
+(the ccwl source, not a CWL file), `gemma_ravanan_manifest`
 (ravanan's own required `--guix-manifest`, a Guix manifest naming
 `gemma-wrapper`, at `cwl/gemma/manifest.scm`) and `ravanan_cache_dir`
 (its own cache directory, separate from `gemma_cache_dir`, for pheno
-files, run logs, and ravanan's own `--store`) are configurable the
-same way as the rest of `src/config.rs`'s layered config.
+files, run logs, the compiled workflow, and ravanan's own `--store`)
+are configurable the same way as the rest of `src/config.rs`'s layered
+config.
 `GET /compute/gemma/status/{run_id}` is shared between both endpoints
 and checks whichever cache directory the `run_id` actually logged to.