# Session log: WiGiTS packaging, from-source Java, CWL with ravanan Work done on this channel in one session: bring the `hmftools` packages up to the tree they are built from, replace every downloaded jar we could compile, document the result, express the pipeline in CWL, and fix the ravanan bugs that blocked it. Everything below is committed. `git log a28bfab..HEAD` in this repo and `git log 16977d6..HEAD` on ravanan's `cwl-directory-and-array-support` branch are the authoritative record. ## Summary | area | outcome | |------------------------|---------| | upstream test baseline | `mvn install -DskipTests` + `mvn -Dmaven.build.cache.enabled=false test`: **3062 tests, 0 failures, 0 errors**, 78 skipped, 36 modules | | channel source pin | `%hmf-source` moved to commit `6531166b66` (amber-4.4 era); 15 module versions bumped to the poms | | shipped jars | `maven-amber` and `maven-bam-tools` are compiled from source instead of downloading the release jars | | dependency tree | 57 of 78 third-party packages compiled from Maven sources jars; 14 published jars and 7 DJL native bundles remain | | README | channel README documents the pin, the package inventory, the from-source machinery and what is not built | | CWL workflows | 15 module tools + 2 helpers, the WGS graph, a QC workflow and a 20-case test suite, all run by ravanan | | ravanan | 6 patches for Directory values, array staging and unlisted outputs; its own 58 unit tests still pass | | artifact | count | notes | |----------------------|-------|-------| | hmftools packages | 41 | 38 modules + 2 `maven-*` variants + `patient-db` library | | third-party packages | 78 | 57 from source, 14 published jars, 7 DJL native bundles | | CWL tools | 17 | 15 module tools, `module-version`, `samtools-index` | | CWL test cases | 20 | 19 by default, `v-chord-e2e` behind `VCHORD=1` | ## 1. Upstream test baseline The `hmftools` checkout is master at `6531166b66` (2026-10-14), which builds as amber 4.4 / purple 4.5 / sage 5.1 / ... although upstream has published no tag for those versions. Baseline before touching anything: ```sh export JAVA_HOME=$(guix build openjdk@17:jdk) export PATH=/tmp/maven/bin:$JAVA_HOME/bin:$PATH # Maven is not packaged cd hmftools mvn -B install -DskipTests # hmf-common test-jar mvn -B -Dmaven.build.cache.enabled=false test ``` Result: **3062 tests, 0 failures, 0 errors, 78 skipped** over 36 modules. `install -DskipTests` has to come first (amber's tests need `hmf-common`'s test-jar), and the build-cache extension must be disabled or a second run silently reuses cached results. ## 2. Channel bump `%hmf-source` now pins commit `6531166b66` (`1nbr2csac8naz5x21wrza4qlhkykzmk6avn977333ysxn3di9h7g`) instead of the `amber-v4.3` tag, and 15 module versions follow the poms: amber 4.4, purple 4.5, cobalt 3.1, esvee 2.1, isofox 2.1, lilac 2.1, linx 2.4, orange 5.1, orange-datamodel 5.1, panel-builder 1.2, redux 2.1, sage 5.1, virus-interpreter 1.7.1, wisp 1.4, bam-tools 1.6.2, plus `hmf-common` 4.4 and `maven-bam-tools` 1.6.2. Four modules moved since `amber-v4.3`: `finding`/`finding-datamodel` were dropped (moved to a separate repo), `fastq-tools`/`health-checker` are pinned to `%hmf-source-4.3` (`hmftools-module` gained `#:source` for this), and `taur` is new (`java-taur`). ## 3. The shipped jars `maven-amber` and `maven-bam-tools` were the last packages that installed a release artifact verbatim (and `maven-amber` was stuck at 4.3 because no `amber_v4.4.jar` exists). Both are now compiled from `%hmf-source` and keep the artifacts' visible behaviour: the names, the `bamtools` alias, and the release launcher that lifts `-Xm*`/`-D*`/`-XX*` out of `argv` into JVM options (`#:launcher-style 'release`). ## 4. The dependency tree `hmf/packages/java-libraries.scm` went from "every dependency is a downloaded jar" to 57 packages compiled here: ```sh guix build -L . --no-offload --keep-going \ $(grep -oP '^\(define-public \Kjava-[a-z0-9.-]+' hmf/packages/java-libraries.scm) ``` The workhorse is `maven-central-source-jar`: fetch `ARTIFACT-VERSION-sources.jar`, compile on openjdk17, install under the same `-.jar` name, and reproduce what Maven does around javac — dropping `module-info.java`, pruning the multi-release `META-INF/versions/` overlay, re-bundling `META-INF/services`, and synthesising `META-INF/maven/.../pom.properties` plus generated resources (`#:extra-resources`). Optional integrations that would drag in unpackaged dependencies are pruned per package (`#:prune`). ## 5. Bugs found on the way | bug | symptom | fix | |--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|-----| | release jars still downloaded | `maven-amber` stuck at 4.3 while the tree builds 4.4 | build both from `%hmf-source`, keep the release launcher and `bamtools` alias | | wrong launcher main classes | `ClassNotFoundException` for cuppa, isofox, neo, pave-reverse, gene-utils, paddle, sigs, sv-tools, compar, bam-tools, teal, cider | use the poms' real classes; library-only modules get no launcher | | `hmf-common` resources missing | `RefGenomeCoordinates` NPE, `ExceptionInInitializerError` in BAM Tools/AMBER/COBALT/PURPLE/SAGE | `bundle-resources` phase jars `src/main/resources` (found by the CWL BAM-metrics step) | | BouncyCastle split | iText 7.1.5 needs the 1.53 API, PDFBox 2/3 need 1.68 | package both lines; they only meet in unused pubkey-encryption code | | DJL metadata | `jnilib/null/...` download attempt, engine version `null` | synthesise `META-INF/maven/.../pom.properties` and `pytorch-engine.properties` | | jOOQ versions | r2dbc API mismatch, missing JAXB/JPA/H2 on codegen classpath | pin `r2dbc-spi 0.9.0.M1`, add the JAXB/JPA/H2 packages, glob the codegen classpath | | H2 sources incomplete | `JSR310Utils` missing from the sources jar | build the git tag and compile `h2/src/main` with `h2/src/java8/src` | | boxable / commons-csv | `CSVFormat.Builder.get()` does not exist in 1.9 | prune boxable's `datatable` helper package (ORANGE uses `BaseTable`) | ## 6. CWL and ravanan `cwl/` describes the WiGiTS pipeline in CWL; ravanan builds one Guix profile from `cwl/manifest.scm` and runs every step in it, so workflow, tools and binaries are pinned by the same commit. ``` cwl/tools/*.cwl 15 module tools + module-version + samtools-index cwl/workflows/wigits.cwl WGS graph: ESVEE/SAGE/PAVE, AMBER/COBALT -> PURPLE -> LINX/LILAC/CHORD/V-CHORD -> ORANGE cwl/workflows/qc.cwl executable two-step workflow (index -> BAM metrics) cwl/tests/ driver, jobs, feature workflows (dir-handoff) ``` ```sh export HMF_GUIX_CHANNEL=$PWD RAVANAN=/path/to/ravanan HMFTOOLS_DIR=/path/to/hmftools \ ./cwl/tests/run-tests.sh # 20/20 with VCHORD=1 ``` The suite covers a `-version` probe for each of the 15 modules, a real PEACH run on the hmftools fixtures, a real two-step workflow, the two ravanan features the graph depends on (`array-stage`, `dir-handoff`), and optionally V-CHORD end to end (breast 0.7854 / ovarian 0.8577 / pancreatic 0.8223 / prostate 0.8063 / other 0.8236). Two of the "ravanan limitations" first written down were my own bugs: an unquoted `File[]` inside a YAML flow mapping is invalid (the brackets end a plain scalar) and guile-libyaml crashes rather than reporting it. The real gaps are the six patches below. ## 7. ravanan patches Branch `cwl-directory-and-array-support` in the ravanan checkout; build it with `guix build -L .guix -f .guix/ravanan-package.scm`. Its own unit suite still passes (`./pre-inst-env guile-run64 tests/*.scm tests/work/*.scm` → **58 passed**, all suites in one process as the Makefile does it). | area | what it fixes | |------------------------------------|---------------| | `reader`, `work/command-line-tool` | `coerce-type` rejected Directory inputs ("Invalid type to coerce to"); canonicalizing a directory hashed it as a file ("Is a directory") | | `command-line-tool` (job script) | input copies, `InitialWorkDirRequirement` and output capture assumed regular files; staging an array raised "Staging array of File objects not implemented yet" | | `work/command-line-tool` | `command-line-binding->args` raised a user-error for Directory inputs, so a tool taking one died before running its command | | `command-line-tool` | `string-null?` is not imported by the job script, so staging an entry with a basename died on an unbound variable | | `store` | interning hashed the path with `file-hash`; directories are now hashed over their (path, checksum) listing and copied into the store | | `workflow` | a tool output the step does not list left a `#f` in the output mapping and surfaced as "Unknown output" | Nothing was pushed or mailed upstream. `Directory` support is the interesting one for other users: any CWL tool that hands a stage directory to the next step needs it. ## 8. Commits This repo (`main`): | commit | subject | |-----------|---------| | `fb2ee4b` | Move the source pin to amber-4.4-era master and bump 15 modules | | `60177a6` | test/.gitignore: ignore work/, not test/work/ | | `385ad5b` | Build maven-amber and maven-bam-tools from source | | `68ff8e0` | Build the Java dependency tree from source instead of shipping jars | | `6864db6` | Add a README documenting the channel, the source builds and their limits | | `5d75a7b` | java-hmf-common: bundle src/main/resources into hmf-common.jar | | `832e286` | Add CWL descriptions of the WiGiTS pipeline, run with ravanan | | `59e289b` | v-chord end-to-end case; document array-staging limitation | | `ced2979` | record how ravanan's own test suites fare here | | `b31e75d` | add the CWL/ravanan work to the README; align all markdown tables | | `65d43a2` | Drop the workarounds now that ravanan handles Directory and arrays | ravanan (`cwl-directory-and-array-support`): | commit | subject | |-----------|---------| | `d65cf05` | reader, work/command-line-tool: treat Directory as a first-class value | | `886fb26` | command-line-tool: copy, stage and capture Directory values and arrays | | `7aef8d9` | store: intern Directory values | | `b079c2f` | workflow: ignore tool outputs that the step does not list | | `9f767ab` | command-line-tool: Avoid string-null?, which the job script does not import | | `b496f85` | work/command-line-tool: Pass Directory inputs on the command line | ## 9. Verification summary | check | command | result | |--------------------|---------------------------------------------------------------|--------| | channel builds | `guix build -L . --no-offload --keep-going ` | exit 0 | | launchers | every `bin/` script started once, no `ClassNotFoundException` | pass | | upstream tests | `mvn -Dmaven.build.cache.enabled=false test` | 3062 tests, 0 failures | | v-chord end to end | `./test/test-vchord.sh` | scores reproduced | | CWL suite | `./cwl/tests/run-tests.sh` (`VCHORD=1`) | 20/20 | | ravanan unit tests | `guile-run64 tests/*.scm tests/work/*.scm` | 58 passed | ## 10. Open items | item | state | |------------------------------------------|-------| | GPU (RTX 5080) from this sandbox | impossible: `/dev` is a private tmpfs with no `nvidia*` nodes, `CapEff=0` so `mknod` is denied, and no `libcuda.so*` exists anywhere (`/`, `/gnu/store`); only the kernel module is visible via `/proc/driver/nvidia` | | GPU with the packaged DJL stack | impossible even on the host: newest DJL native is `pytorch-native-cu124` 2.5.1 (CUDA 12.4 -> `sm_90`), the card is `sm_120`; our cu121 `libtorch_cuda.so` lists `sm_35`…`sm_90` only | | GPU, plausible route (host only) | DJL 0.39 (`pytorch-jni` 2.7.1, all 17 DJL classes used are unchanged) + system libtorch 2.7.1+cu128 via `PYTORCH_LIBRARY_PATH`; needs the device visible and a ~2.5 GB download | | `workflows/wigits.cwl` execution | not run end to end: needs an Ensembl cache, GC profile, BAF loci VCF, LILAC resources and indexed BAMs; `qc.cwl` and `dir-handoff.cwl` are the executable proofs | | ravanan cwltest e2e / conformance suites | not buildable here: `cwltest` -> `python-cheroot` fails two SSL assertions on this OpenSSL | | ravanan branch | `cwl-directory-and-array-support`, 6 commits local; not merged, not sent upstream | ## 11. GPU investigation The host has an NVIDIA GeForce RTX 5080 (driver 610.43.02, open kernel module), visible through `/proc/driver/nvidia`. Two independent blockers prevent using it: | blocker | evidence | |------------------------------------------------|----------| | no device access from the sandbox | `/dev` is a private tmpfs carrying only `null`, `zero`, `full`, `random`, `urandom`, `tty`, `fuse`; `CapEff=0` so `mknod` fails with `EPERM`; no `libcuda.so*` under `/`, `/gnu/store` or `/usr/lib` | | the card needs CUDA 12.8+, DJL ships CUDA 12.4 | newest DJL artifact is `pytorch-native-cu124` 2.5.1; `strings libtorch_cuda.so` from the 2.2.2/cu121 bundle lists `sm_35`…`sm_90` and no `sm_100`/`sm_120`, so a Blackwell card fails with "no kernel image is available for execution on the device" | What would be needed to go further, in order of cost: 1. run `test/test-vchord.sh` on the host, where `/dev/nvidia*` and `/usr/lib/x86_64-linux-gnu/libcuda.so.1` exist — still fails on the second blocker; 2. bump the channel's DJL stack to 0.39 (the API is compatible: all 17 DJL classes v-chord and v-cuppa import still exist), install `pytorch-jni` 2.7.1-0.39.0, and point `PYTORCH_LIBRARY_PATH` at a libtorch 2.7.1+cu128 build from download.pytorch.org (~2.5 GB). That is the first combination that could execute on this card, and it needs a machine with the device visible; 3. or bind-mount `/dev/nvidia*` and the host's `libcuda.so.1` into the sandbox — only the host owner can do that, and it does not help until step 2 is done. Until then the CPU leg of `test/test-vchord.sh` and `v-chord-e2e` in the CWL suite remain the GPU-less proofs.