WiGiTS in CWL
Common Workflow Language descriptions of Hartwig Medical Foundation's WiGiTS pipeline, with ravanan as the runner. Tool binaries come from this channel's Guix packages, so a workflow, its tools and the packages are pinned by one commit.
cwl/
manifest.scm Guix manifest: the WiGiTS tools (this channel)
tools/*.cwl one CommandLineTool per hmftools module
workflows/wigits.cwl the reference WGS graph
workflows/qc.cwl minimal two-step workflow (BAM index -> metrics)
tests/run-tests.sh test driver
tests/jobs/*.yaml job documents (fixtures from the hmftools checkout)
Tools
| CWL tool | command | reads -> writes |
|---|---|---|
amber.cwl |
amber |
BAM pair + loci VCF -> BAF/contamination/SNP calls |
bam-metrics.cwl |
bam-tools |
BAM (+ index) -> <sample>.bam_metric.summary.tsv |
chord.cwl |
chord |
PURPLE/Pave artifacts -> HRD prediction |
cobalt.cwl |
cobalt |
BAM pair + GC profile -> read-depth ratios |
cuppa.cwl |
cuppa |
PURPLE/LINX/ISOFOX artifacts + classifier -> tissue of origin |
esvee.cwl |
esvee |
BAM pair -> structural variants |
isofox.cwl |
isofox |
RNA BAM -> expression, fusions, splice junctions |
lilac.cwl |
lilac |
BAM pair + HLA resources -> HLA types |
linx.cwl |
linx |
PURPLE -> clusters, fusions, drivers |
orange.cwl |
orange |
all of the above -> PDF report + JSON |
pave.cwl |
pave |
SAGE VCF -> annotated VCF |
peach.cwl |
peach |
germline VCF + haplotype TSVs -> phenotypes |
purple.cwl |
purple |
AMBER + COBALT + Pave/Esvee -> CN, purity, ploidy |
sage.cwl |
sage |
BAM pair -> somatic VCF |
v-chord.cwl |
v-chord |
PURPLE + TorchScript model -> HRD scores |
module-version.cwl |
bash -c |
test helper: runs <module> -version |
samtools-index.cwl |
samtools index |
BAM -> .bai |
Stage hand-offs are Directory values in both directions (AMBER/Cobalt
-> PURPLE, PURPLE -> LINX/LILAC/CHORD/ORANGE), which is also what the
hmftools modules expect on their -<stage>_dir flags.
Running
# one tool
ravanan --store=store --outdir=out --guix-manifest=cwl/manifest.scm \
cwl/tools/peach.cwl job.yaml
# the test suite
RAVANAN=/path/to/ravanan HMFTOOLS_DIR=/path/to/hmftools ./cwl/tests/run-tests.sh
manifest.scm needs the channel checkout on Guile's load path and is
loaded with load (where current-filename is #f), so set
export HMF_GUIX_CHANNEL=/path/to/guix-hmf
RAVANAN matters: use ravanan v0.2.0-92 or newer (git describe in
the ravanan checkout). The 0.2.0 release crashes on any tool without an
arguments field (vector-map ... #f); the fix is commit 16977d6 and
later. Build it from the checkout with
guix build -L .guix -f .guix/ravanan-package.scm
What the tests cover
cwl/tests/run-tests.sh runs 17 cases in a few minutes, 18 with
VCHORD=1:
| case | what it proves (all cases pass) |
|---|---|
version-<module> (15) |
tools/module-version.cwl runs <module> -version for every module, so the manifest really exposes it |
peach-haplotypes |
a real PEACH run on the hmftools fixtures (variants.vcf + haplotypes/function/drug TSVs), producing the five .peach.*.tsv outputs |
bam-metrics |
a real two-step workflow: samtools index, then BAM Tools' metrics on a COBALT fixture BAM |
array-stage |
InitialWorkDirRequirement listing a File[] input stages every element into the working directory |
dir-handoff |
one step's Directory output feeds another step's Directory input unchanged |
v-chord-e2e (VCHORD=1) |
the PyTorch HRD classifier in a ravanan job on the 45 MB TSO500 model; reproduces test/test-vchord.sh's scores (breast 0.7854, ovarian 0.8577, ...) |
The packaging probes are not ceremonial: they are how java-sage and
java-purple were found missing from the manifest, and how the missing
src/main/resources in hmf-common.jar surfaced (it makes
RefGenomeCoordinates NPE, which takes down BAM Tools, AMBER, COBALT,
PURPLE and SAGE at runtime).
ravanan's own test suites
Run from the ravanan checkout, with a development profile built as
;; guix build -L .guix -f dev-profile.scm
(profile (allow-collisions? #t)
(content (packages->manifest
(append (map manifest-entry-item
(manifest-entries
(package->development-manifest ravanan)))
(list node guile-run64 ccwl)))))
| suite | command | result |
|---|---|---|
| unit tests | ./pre-inst-env guile-run64 tests/*.scm tests/work/*.scm |
58 passed |
e2e (.guix/e2e-tests.scm) |
needs cwltest |
not buildable here (see below) |
CWL conformance (.guix/cwl-conformance.scm) |
needs cwltest |
not buildable here (see below) |
The unit suites must run in one process, as the Makefile does:
tests/propnet.scm uses state-let* without importing
(ravanan work monads) and only sees it because an earlier suite pulls
the module in.
The cwltest-based suites cannot be built here: they need cwltest,
whose dependency python-cheroot fails two SSL assertions on this
OpenSSL (test_https_over_http_error: the error text is
"WRONG_VERSION_NUMBER", the test expects "record layer failure").
--without-tests=python-cheroot does not reach dependencies, and a
package-mapping that disables cheroot's tests did not propagate
through cwl-runner, so this needs a patch to cheroot's test suite.
cwl/tests/ covers the same ground for this channel.
Graph
workflows/wigits.cwl is the WGS tumor/reference graph:
ESVEE ──┐
SAGE ───┴─> PAVE ──┐
AMBER ─────────────┼─> PURPLE ─┬─> LINX ──┐
COBALT ────────────┘ ├─> LILAC ─┼─> ORANGE
├─> CHORD ─┤
└─> V-CHORD┘
RNA stages live in separate workflows (ISOFOX/CUPPA take a different
sample's BAM). The graph needs real reference data to execute: an
Ensembl data cache, a GC profile, a BAF loci VCF, a LILAC resource
directory and indexed BAMs, so it is not part of the automatic suite —
workflows/qc.cwl is the executable end-to-end proof.
ravanan patches these workflows need
Upstream ravanan at v0.2.0-92 cannot run Directory-based workflows. Six
patches on branch cwl-directory-and-array-support of the ravanan
checkout fix that; build it with
guix build -L .guix -f .guix/ravanan-package.scm # in the ravanan checkout
| area | what it fixes |
|---|---|
reader, work/command-line-tool |
coerce-type rejected Directory inputs ("Invalid type to coerce to"); canonicalizing a directory hashed it as a file ("Is a directory") |
command-line-tool (script) |
input copies, InitialWorkDirRequirement and output capture assumed regular files; staging an array raised "Staging array of File objects not implemented yet" |
work/command-line-tool |
command-line-binding->args raised a user-error for Directory inputs, so a tool taking one died before running |
command-line-tool |
avoided string-null?, which the job script cannot import |
store |
interning hashed the path with file-hash; directories are now hashed over their (path, checksum) listing and copied in |
workflow |
a tool output the step does not list left a #f in the output mapping and surfaced as "Unknown output" |
All 58 of ravanan's own unit tests still pass with these applied; the six commits are local to that checkout (nothing was pushed upstream).
One libyaml trap is worth knowing independent of ravanan: in a YAML flow
mapping, an unquoted File[] is invalid (the brackets terminate a plain
scalar), and guile-libyaml crashes on it (cdata-sel: attempt to
dereference null pointer) instead of reporting a parse error. Write
array types quoted — type: "File[]" — or use block style.
Notes on individual modules
- PATH policy: ravanan refuses to look up executables on
PATH, so every command must come from the manifest.manifest.scmtherefore lists one package per module plusbash,coreutilsandsamtools. -ref_genometakes a file path for the modules here (BAM Tools, SAGE, ESVEE, LILAC, ISOFOX) and needs a matching.fai.- ORANGE's PDF backend needs iText (see the channel's
java-orange); its-circosinput is not packaged, so plots are skipped unless a circos binary is supplied. - CUPPA needs a Python interpreter and a classifier pickle: both are
explicit
string/Fileinputs rather than hidden dependencies.
