diff options
| author | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-10-19 07:13:15 +0000 |
|---|---|---|
| committer | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-10-19 07:13:15 +0000 |
| commit | d504b5da9f5deeece9ba2ff4dcd8baf3dfa4dc32 (patch) | |
| tree | eaed193876da61ad7d6ae55906dccf4c6198c583 /README.md | |
| parent | 477294a2aa9ae7348137271b66831ac93a6149d6 (diff) | |
| download | genenetwork4-gemma-cwl-parallel.tar.gz | |
gemma: compile ravanan's ccwl workflow on demand, don't check in the CWL gemma-cwl-parallel
gemma-loco.cwl was a generated artifact (ccwl compile's output), but was being committed like hand-written source. Remove it, and have compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL file under its own cache dir on every call instead -- cheap (pure macro expansion, no Guix build involved), so there's no real cost to regenerating it fresh each time, and no risk of a stale compiled file drifting from its ccwl source. Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH, alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now names the .scm source, not a .cwl file. Verified end to end by running the exact compile-then-run sequence compute_ravanan now performs against the real ccwl/ravanan/gemma-wrapper binaries and the BXD fixtures: same correctness and timings as before.
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 37 |
1 files changed, 21 insertions, 16 deletions
diff --git a/README.md b/README.md index 5c5719f..421e56f 100644 --- a/README.md +++ b/README.md @@ -1002,32 +1002,37 @@ yet) is a 404, not an empty response -- genuinely different from Same request/response shape as `/compute/gemma`, same two gemma-wrapper steps (kinship, then GWA, both `--loco`), but driven by -[ravanan](https://github.com/vepiteski/ravanan) running the CWL -workflow at `cwl/gemma/gemma-loco.cwl` instead of this server's own +[ravanan](https://github.com/vepiteski/ravanan) running a CWL workflow +compiled on every call (via `ccwl compile`, see below) from the ccwl +source at `cwl/gemma/gemma-loco.scm`, instead of this server's own direct subprocess calls -- an alternative runner for the same computation, kept side by side with `/compute/gemma` so the two can be compared (correctness, timing). Unlike gemma-wrapper's own `--loco` mode, which fans its per-chromosome kinship/GWA jobs out internally via GNU `parallel`, this workflow has ravanan itself do that fan-out: it scatters one `gemma-wrapper --chromosomes <chr> --no-parallel ...` -invocation per chromosome (written in +invocation per chromosome, each restricted to exactly the chromosome +it was given (gemma-wrapper itself still does the correct "exclude +this chromosome from kinship, test only its markers" LOCO logic, +against the *full* genotype/SNP files -- `--chromosomes` only +overrides which chromosomes gemma-wrapper loops over, not the +per-chromosome math itself). This workflow is written in [ccwl](https://ccwl.systemreboot.net/) at `cwl/gemma/gemma-loco.scm`, -compiled to `gemma-loco.cwl` with `ccwl compile`), each restricted to -exactly the chromosome it was -given (gemma-wrapper itself still does the correct "exclude this -chromosome from kinship, test only its markers" LOCO logic, against -the *full* genotype/SNP files -- `--chromosomes` only overrides which -chromosomes gemma-wrapper loops over, not the per-chromosome math -itself). `compute_ravanan` computes the chromosome list from -`snps_file` itself (gemma-wrapper's own `get_chromosomes`, reproduced -as `chromosomes_in`) and passes it as the workflow's `chromosomes` -input, since CWL's `scatter` needs a concrete array at invocation -time. `ravanan_cmd`, `gemma_ravanan_workflow`, `gemma_ravanan_manifest` +not raw CWL; `compute_ravanan` compiles it to CWL with `ccwl compile` +fresh on every call (cheap -- pure macro expansion, no Guix build +involved) into a generated, throwaway file under `ravanan_cache_dir`, +never checked into the repo. `compute_ravanan` computes the chromosome +list from `snps_file` itself (gemma-wrapper's own `get_chromosomes`, +reproduced as `chromosomes_in`) and passes it as the workflow's +`chromosomes` input, since CWL's `scatter` needs a concrete array at +invocation time. `ravanan_cmd`, `ccwl_cmd`, `gemma_ravanan_workflow` +(the ccwl source, not a CWL file), `gemma_ravanan_manifest` (ravanan's own required `--guix-manifest`, a Guix manifest naming `gemma-wrapper`, at `cwl/gemma/manifest.scm`) and `ravanan_cache_dir` (its own cache directory, separate from `gemma_cache_dir`, for pheno -files, run logs, and ravanan's own `--store`) are configurable the -same way as the rest of `src/config.rs`'s layered config. +files, run logs, the compiled workflow, and ravanan's own `--store`) +are configurable the same way as the rest of `src/config.rs`'s layered +config. `GET /compute/gemma/status/{run_id}` is shared between both endpoints and checks whichever cache directory the `run_id` actually logged to. |
