about summary refs log tree commit diff

WiGiTS in CWL

Common Workflow Language descriptions of Hartwig Medical Foundation's WiGiTS pipeline, with ravanan as the runner. Tool binaries come from this channel's Guix packages, so a workflow, its tools and the packages are pinned by one commit.

cwl/
  manifest.scm            Guix manifest: the WiGiTS tools (this channel)
  tools/*.cwl             one CommandLineTool per hmftools module
  workflows/wigits.cwl    the reference WGS graph
  workflows/qc.cwl        minimal two-step workflow (BAM index -> metrics)
  tests/run-tests.sh      test driver
  tests/jobs/*.yaml       job documents (fixtures from the hmftools checkout)

Tools

CWL tool command reads -> writes
amber.cwl amber BAM pair + loci VCF -> BAF/contamination/SNP calls
bam-metrics.cwl bam-tools BAM (+ index) -> <sample>.bam_metric.summary.tsv
chord.cwl chord PURPLE/Pave artifacts -> HRD prediction
cobalt.cwl cobalt BAM pair + GC profile -> read-depth ratios
cuppa.cwl cuppa PURPLE/LINX/ISOFOX artifacts + classifier -> tissue of origin
esvee.cwl esvee BAM pair -> structural variants
isofox.cwl isofox RNA BAM -> expression, fusions, splice junctions
lilac.cwl lilac BAM pair + HLA resources -> HLA types
linx.cwl linx PURPLE -> clusters, fusions, drivers
orange.cwl orange all of the above -> PDF report + JSON
pave.cwl pave SAGE VCF -> annotated VCF
peach.cwl peach germline VCF + haplotype TSVs -> phenotypes
purple.cwl purple AMBER + COBALT + Pave/Esvee -> CN, purity, ploidy
sage.cwl sage BAM pair -> somatic VCF
v-chord.cwl v-chord PURPLE + TorchScript model -> HRD scores
module-version.cwl bash -c test helper: runs <module> -version
samtools-index.cwl samtools index BAM -> .bai

Stage hand-offs are Directory values in both directions (AMBER/Cobalt -> PURPLE, PURPLE -> LINX/LILAC/CHORD/ORANGE), which is also what the hmftools modules expect on their -<stage>_dir flags.

Running

# one tool
ravanan --store=store --outdir=out --guix-manifest=cwl/manifest.scm \
        cwl/tools/peach.cwl job.yaml

# the test suite
RAVANAN=/path/to/ravanan HMFTOOLS_DIR=/path/to/hmftools ./cwl/tests/run-tests.sh

manifest.scm needs the channel checkout on Guile's load path and is loaded with load (where current-filename is #f), so set

export HMF_GUIX_CHANNEL=/path/to/guix-hmf

RAVANAN matters: use ravanan v0.2.0-92 or newer (git describe in the ravanan checkout). The 0.2.0 release crashes on any tool without an arguments field (vector-map ... #f); the fix is commit 16977d6 and later. Build it from the checkout with

guix build -L .guix -f .guix/ravanan-package.scm

What the tests cover

cwl/tests/run-tests.sh runs 17 cases in a few minutes, 18 with VCHORD=1:

case what it proves (all cases pass)
version-<module> (15) tools/module-version.cwl runs <module> -version for every module, so the manifest really exposes it
peach-haplotypes a real PEACH run on the hmftools fixtures (variants.vcf + haplotypes/function/drug TSVs), producing the five .peach.*.tsv outputs
bam-metrics a real two-step workflow: samtools index, then BAM Tools' metrics on a COBALT fixture BAM
array-stage InitialWorkDirRequirement listing a File[] input stages every element into the working directory
dir-handoff one step's Directory output feeds another step's Directory input unchanged
v-chord-e2e (VCHORD=1) the PyTorch HRD classifier in a ravanan job on the 45 MB TSO500 model; reproduces test/test-vchord.sh's scores (breast 0.7854, ovarian 0.8577, ...)

The packaging probes are not ceremonial: they are how java-sage and java-purple were found missing from the manifest, and how the missing src/main/resources in hmf-common.jar surfaced (it makes RefGenomeCoordinates NPE, which takes down BAM Tools, AMBER, COBALT, PURPLE and SAGE at runtime).

ravanan's own test suites

Run from the ravanan checkout, with a development profile built as

;; guix build -L .guix -f dev-profile.scm
(profile (allow-collisions? #t)
         (content (packages->manifest
                   (append (map manifest-entry-item
                                (manifest-entries
                                 (package->development-manifest ravanan)))
                           (list node guile-run64 ccwl)))))
suite command result
unit tests ./pre-inst-env guile-run64 tests/*.scm tests/work/*.scm 58 passed
e2e (.guix/e2e-tests.scm) needs cwltest not buildable here (see below)
CWL conformance (.guix/cwl-conformance.scm) needs cwltest not buildable here (see below)

The unit suites must run in one process, as the Makefile does: tests/propnet.scm uses state-let* without importing (ravanan work monads) and only sees it because an earlier suite pulls the module in.

The cwltest-based suites cannot be built here: they need cwltest, whose dependency python-cheroot fails two SSL assertions on this OpenSSL (test_https_over_http_error: the error text is "WRONG_VERSION_NUMBER", the test expects "record layer failure"). --without-tests=python-cheroot does not reach dependencies, and a package-mapping that disables cheroot's tests did not propagate through cwl-runner, so this needs a patch to cheroot's test suite. cwl/tests/ covers the same ground for this channel.

Graph

workflows/wigits.cwl is the WGS tumor/reference graph:

ESVEE ──┐
SAGE ───┴─> PAVE ──┐
AMBER ─────────────┼─> PURPLE ─┬─> LINX ──┐
COBALT ────────────┘           ├─> LILAC ─┼─> ORANGE
                               ├─> CHORD ─┤
                               └─> V-CHORD┘

RNA stages live in separate workflows (ISOFOX/CUPPA take a different sample's BAM). The graph needs real reference data to execute: an Ensembl data cache, a GC profile, a BAF loci VCF, a LILAC resource directory and indexed BAMs, so it is not part of the automatic suite — workflows/qc.cwl is the executable end-to-end proof.

ravanan patches these workflows need

Upstream ravanan at v0.2.0-92 cannot run Directory-based workflows. Six patches on branch cwl-directory-and-array-support of the ravanan checkout fix that; build it with

guix build -L .guix -f .guix/ravanan-package.scm     # in the ravanan checkout
area what it fixes
reader, work/command-line-tool coerce-type rejected Directory inputs ("Invalid type to coerce to"); canonicalizing a directory hashed it as a file ("Is a directory")
command-line-tool (script) input copies, InitialWorkDirRequirement and output capture assumed regular files; staging an array raised "Staging array of File objects not implemented yet"
work/command-line-tool command-line-binding->args raised a user-error for Directory inputs, so a tool taking one died before running
command-line-tool avoided string-null?, which the job script cannot import
store interning hashed the path with file-hash; directories are now hashed over their (path, checksum) listing and copied in
workflow a tool output the step does not list left a #f in the output mapping and surfaced as "Unknown output"

All 58 of ravanan's own unit tests still pass with these applied; the six commits are local to that checkout (nothing was pushed upstream).

One libyaml trap is worth knowing independent of ravanan: in a YAML flow mapping, an unquoted File[] is invalid (the brackets terminate a plain scalar), and guile-libyaml crashes on it (cdata-sel: attempt to dereference null pointer) instead of reporting a parse error. Write array types quoted — type: "File[]" — or use block style.

Notes on individual modules

  • PATH policy: ravanan refuses to look up executables on PATH, so every command must come from the manifest. manifest.scm therefore lists one package per module plus bash, coreutils and samtools.
  • -ref_genome takes a file path for the modules here (BAM Tools, SAGE, ESVEE, LILAC, ISOFOX) and needs a matching .fai.
  • ORANGE's PDF backend needs iText (see the channel's java-orange); its -circos input is not packaged, so plots are skipped unless a circos binary is supplied.
  • CUPPA needs a Python interpreter and a classifier pickle: both are explicit string/File inputs rather than hidden dependencies.