1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
|
# Session log: WiGiTS packaging, from-source Java, CWL with ravanan
Work done on this channel in one session: bring the `hmftools` packages up
to the tree they are built from, replace every downloaded jar we could
compile, document the result, express the pipeline in CWL, and fix the
ravanan bugs that blocked it.
Everything below is committed. `git log a28bfab..HEAD` in this repo and
`git log 16977d6..HEAD` on ravanan's `cwl-directory-and-array-support`
branch are the authoritative record.
## Summary
| area | outcome |
|------------------------|---------|
| upstream test baseline | `mvn install -DskipTests` + `mvn -Dmaven.build.cache.enabled=false test`: **3062 tests, 0 failures, 0 errors**, 78 skipped, 36 modules |
| channel source pin | `%hmf-source` moved to commit `6531166b66` (amber-4.4 era); 15 module versions bumped to the poms |
| shipped jars | `maven-amber` and `maven-bam-tools` are compiled from source instead of downloading the release jars |
| dependency tree | 57 of 78 third-party packages compiled from Maven sources jars; 14 published jars and 7 DJL native bundles remain |
| README | channel README documents the pin, the package inventory, the from-source machinery and what is not built |
| CWL workflows | 15 module tools + 2 helpers, the WGS graph, a QC workflow and a 20-case test suite, all run by ravanan |
| ravanan | 6 patches for Directory values, array staging and unlisted outputs; its own 58 unit tests still pass |
| artifact | count | notes |
|----------------------|-------|-------|
| hmftools packages | 41 | 38 modules + 2 `maven-*` variants + `patient-db` library |
| third-party packages | 78 | 57 from source, 14 published jars, 7 DJL native bundles |
| CWL tools | 17 | 15 module tools, `module-version`, `samtools-index` |
| CWL test cases | 20 | 19 by default, `v-chord-e2e` behind `VCHORD=1` |
## 1. Upstream test baseline
The `hmftools` checkout is master at `6531166b66` (2026-10-14), which
builds as amber 4.4 / purple 4.5 / sage 5.1 / ... although upstream has
published no tag for those versions. Baseline before touching anything:
```sh
export JAVA_HOME=$(guix build openjdk@17:jdk)
export PATH=/tmp/maven/bin:$JAVA_HOME/bin:$PATH # Maven is not packaged
cd hmftools
mvn -B install -DskipTests # hmf-common test-jar
mvn -B -Dmaven.build.cache.enabled=false test
```
Result: **3062 tests, 0 failures, 0 errors, 78 skipped** over 36 modules.
`install -DskipTests` has to come first (amber's tests need
`hmf-common`'s test-jar), and the build-cache extension must be disabled
or a second run silently reuses cached results.
## 2. Channel bump
`%hmf-source` now pins commit `6531166b66`
(`1nbr2csac8naz5x21wrza4qlhkykzmk6avn977333ysxn3di9h7g`) instead of the
`amber-v4.3` tag, and 15 module versions follow the poms: amber 4.4,
purple 4.5, cobalt 3.1, esvee 2.1, isofox 2.1, lilac 2.1, linx 2.4,
orange 5.1, orange-datamodel 5.1, panel-builder 1.2, redux 2.1, sage 5.1,
virus-interpreter 1.7.1, wisp 1.4, bam-tools 1.6.2, plus `hmf-common` 4.4
and `maven-bam-tools` 1.6.2.
Four modules moved since `amber-v4.3`: `finding`/`finding-datamodel` were
dropped (moved to a separate repo), `fastq-tools`/`health-checker` are
pinned to `%hmf-source-4.3` (`hmftools-module` gained `#:source` for
this), and `taur` is new (`java-taur`).
## 3. The shipped jars
`maven-amber` and `maven-bam-tools` were the last packages that installed
a release artifact verbatim (and `maven-amber` was stuck at 4.3 because
no `amber_v4.4.jar` exists). Both are now compiled from `%hmf-source`
and keep the artifacts' visible behaviour: the names, the `bamtools`
alias, and the release launcher that lifts `-Xm*`/`-D*`/`-XX*` out of
`argv` into JVM options (`#:launcher-style 'release`).
## 4. The dependency tree
`hmf/packages/java-libraries.scm` went from "every dependency is a
downloaded jar" to 57 packages compiled here:
```sh
guix build -L . --no-offload --keep-going \
$(grep -oP '^\(define-public \Kjava-[a-z0-9.-]+' hmf/packages/java-libraries.scm)
```
The workhorse is `maven-central-source-jar`: fetch
`ARTIFACT-VERSION-sources.jar`, compile on openjdk17, install under the
same `<artifact>-<version>.jar` name, and reproduce what Maven does around
javac — dropping `module-info.java`, pruning the multi-release
`META-INF/versions/` overlay, re-bundling `META-INF/services`, and
synthesising `META-INF/maven/.../pom.properties` plus generated resources
(`#:extra-resources`). Optional integrations that would drag in
unpackaged dependencies are pruned per package (`#:prune`).
## 5. Bugs found on the way
| bug | symptom | fix |
|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|-----|
| release jars still downloaded | `maven-amber` stuck at 4.3 while the tree builds 4.4 | build both from `%hmf-source`, keep the release launcher and `bamtools` alias |
| wrong launcher main classes | `ClassNotFoundException` for cuppa, isofox, neo, pave-reverse, gene-utils, paddle, sigs, sv-tools, compar, bam-tools, teal, cider | use the poms' real classes; library-only modules get no launcher |
| `hmf-common` resources missing | `RefGenomeCoordinates` NPE, `ExceptionInInitializerError` in BAM Tools/AMBER/COBALT/PURPLE/SAGE | `bundle-resources` phase jars `src/main/resources` (found by the CWL BAM-metrics step) |
| BouncyCastle split | iText 7.1.5 needs the 1.53 API, PDFBox 2/3 need 1.68 | package both lines; they only meet in unused pubkey-encryption code |
| DJL metadata | `jnilib/null/...` download attempt, engine version `null` | synthesise `META-INF/maven/.../pom.properties` and `pytorch-engine.properties` |
| jOOQ versions | r2dbc API mismatch, missing JAXB/JPA/H2 on codegen classpath | pin `r2dbc-spi 0.9.0.M1`, add the JAXB/JPA/H2 packages, glob the codegen classpath |
| H2 sources incomplete | `JSR310Utils` missing from the sources jar | build the git tag and compile `h2/src/main` with `h2/src/java8/src` |
| boxable / commons-csv | `CSVFormat.Builder.get()` does not exist in 1.9 | prune boxable's `datatable` helper package (ORANGE uses `BaseTable`) |
## 6. CWL and ravanan
`cwl/` describes the WiGiTS pipeline in CWL; ravanan builds one Guix
profile from `cwl/manifest.scm` and runs every step in it, so workflow,
tools and binaries are pinned by the same commit.
```
cwl/tools/*.cwl 15 module tools + module-version + samtools-index
cwl/workflows/wigits.cwl WGS graph: ESVEE/SAGE/PAVE, AMBER/COBALT -> PURPLE
-> LINX/LILAC/CHORD/V-CHORD -> ORANGE
cwl/workflows/qc.cwl executable two-step workflow (index -> BAM metrics)
cwl/tests/ driver, jobs, feature workflows (dir-handoff)
```
```sh
export HMF_GUIX_CHANNEL=$PWD
RAVANAN=/path/to/ravanan HMFTOOLS_DIR=/path/to/hmftools \
./cwl/tests/run-tests.sh # 20/20 with VCHORD=1
```
The suite covers a `-version` probe for each of the 15 modules, a real
PEACH run on the hmftools fixtures, a real two-step workflow, the two
ravanan features the graph depends on (`array-stage`, `dir-handoff`), and
optionally V-CHORD end to end (breast 0.7854 / ovarian 0.8577 /
pancreatic 0.8223 / prostate 0.8063 / other 0.8236).
Two of the "ravanan limitations" first written down were my own bugs: an
unquoted `File[]` inside a YAML flow mapping is invalid (the brackets end
a plain scalar) and guile-libyaml crashes rather than reporting it. The
real gaps are the six patches below.
## 7. ravanan patches
Branch `cwl-directory-and-array-support` in the ravanan checkout; build it
with `guix build -L .guix -f .guix/ravanan-package.scm`. Its own unit
suite still passes (`./pre-inst-env guile-run64 tests/*.scm
tests/work/*.scm` → **58 passed**, all suites in one process as the
Makefile does it).
| area | what it fixes |
|------------------------------------|---------------|
| `reader`, `work/command-line-tool` | `coerce-type` rejected Directory inputs ("Invalid type to coerce to"); canonicalizing a directory hashed it as a file ("Is a directory") |
| `command-line-tool` (job script) | input copies, `InitialWorkDirRequirement` and output capture assumed regular files; staging an array raised "Staging array of File objects not implemented yet" |
| `work/command-line-tool` | `command-line-binding->args` raised a user-error for Directory inputs, so a tool taking one died before running its command |
| `command-line-tool` | `string-null?` is not imported by the job script, so staging an entry with a basename died on an unbound variable |
| `store` | interning hashed the path with `file-hash`; directories are now hashed over their (path, checksum) listing and copied into the store |
| `workflow` | a tool output the step does not list left a `#f` in the output mapping and surfaced as "Unknown output" |
Nothing was pushed or mailed upstream. `Directory` support is the
interesting one for other users: any CWL tool that hands a stage
directory to the next step needs it.
## 8. Commits
This repo (`main`):
| commit | subject |
|-----------|---------|
| `fb2ee4b` | Move the source pin to amber-4.4-era master and bump 15 modules |
| `60177a6` | test/.gitignore: ignore work/, not test/work/ |
| `385ad5b` | Build maven-amber and maven-bam-tools from source |
| `68ff8e0` | Build the Java dependency tree from source instead of shipping jars |
| `6864db6` | Add a README documenting the channel, the source builds and their limits |
| `5d75a7b` | java-hmf-common: bundle src/main/resources into hmf-common.jar |
| `832e286` | Add CWL descriptions of the WiGiTS pipeline, run with ravanan |
| `59e289b` | v-chord end-to-end case; document array-staging limitation |
| `ced2979` | record how ravanan's own test suites fare here |
| `b31e75d` | add the CWL/ravanan work to the README; align all markdown tables |
| `65d43a2` | Drop the workarounds now that ravanan handles Directory and arrays |
ravanan (`cwl-directory-and-array-support`):
| commit | subject |
|-----------|---------|
| `d65cf05` | reader, work/command-line-tool: treat Directory as a first-class value |
| `886fb26` | command-line-tool: copy, stage and capture Directory values and arrays |
| `7aef8d9` | store: intern Directory values |
| `b079c2f` | workflow: ignore tool outputs that the step does not list |
| `9f767ab` | command-line-tool: Avoid string-null?, which the job script does not import |
| `b496f85` | work/command-line-tool: Pass Directory inputs on the command line |
## 9. Verification summary
| check | command | result |
|--------------------|---------------------------------------------------------------|--------|
| channel builds | `guix build -L . --no-offload --keep-going <all packages>` | exit 0 |
| launchers | every `bin/` script started once, no `ClassNotFoundException` | pass |
| upstream tests | `mvn -Dmaven.build.cache.enabled=false test` | 3062 tests, 0 failures |
| v-chord end to end | `./test/test-vchord.sh` | scores reproduced |
| CWL suite | `./cwl/tests/run-tests.sh` (`VCHORD=1`) | 20/20 |
| ravanan unit tests | `guile-run64 tests/*.scm tests/work/*.scm` | 58 passed |
## 10. Open items
| item | state |
|------------------------------------------|-------|
| GPU (RTX 5080) from this sandbox | impossible: `/dev` is a private tmpfs with no `nvidia*` nodes, `CapEff=0` so `mknod` is denied, and no `libcuda.so*` exists anywhere (`/`, `/gnu/store`); only the kernel module is visible via `/proc/driver/nvidia` |
| GPU with the packaged DJL stack | impossible even on the host: newest DJL native is `pytorch-native-cu124` 2.5.1 (CUDA 12.4 -> `sm_90`), the card is `sm_120`; our cu121 `libtorch_cuda.so` lists `sm_35`…`sm_90` only |
| GPU, plausible route (host only) | DJL 0.39 (`pytorch-jni` 2.7.1, all 17 DJL classes used are unchanged) + system libtorch 2.7.1+cu128 via `PYTORCH_LIBRARY_PATH`; needs the device visible and a ~2.5 GB download |
| `workflows/wigits.cwl` execution | not run end to end: needs an Ensembl cache, GC profile, BAF loci VCF, LILAC resources and indexed BAMs; `qc.cwl` and `dir-handoff.cwl` are the executable proofs |
| ravanan cwltest e2e / conformance suites | not buildable here: `cwltest` -> `python-cheroot` fails two SSL assertions on this OpenSSL |
| ravanan branch | `cwl-directory-and-array-support`, 6 commits local; not merged, not sent upstream |
## 11. GPU investigation
The host has an NVIDIA GeForce RTX 5080 (driver 610.43.02, open kernel
module), visible through `/proc/driver/nvidia`. Two independent blockers
prevent using it:
| blocker | evidence |
|------------------------------------------------|----------|
| no device access from the sandbox | `/dev` is a private tmpfs carrying only `null`, `zero`, `full`, `random`, `urandom`, `tty`, `fuse`; `CapEff=0` so `mknod` fails with `EPERM`; no `libcuda.so*` under `/`, `/gnu/store` or `/usr/lib` |
| the card needs CUDA 12.8+, DJL ships CUDA 12.4 | newest DJL artifact is `pytorch-native-cu124` 2.5.1; `strings libtorch_cuda.so` from the 2.2.2/cu121 bundle lists `sm_35`…`sm_90` and no `sm_100`/`sm_120`, so a Blackwell card fails with "no kernel image is available for execution on the device" |
What would be needed to go further, in order of cost:
1. run `test/test-vchord.sh` on the host, where `/dev/nvidia*` and
`/usr/lib/x86_64-linux-gnu/libcuda.so.1` exist — still fails on the
second blocker;
2. bump the channel's DJL stack to 0.39 (the API is compatible: all 17
DJL classes v-chord and v-cuppa import still exist), install
`pytorch-jni` 2.7.1-0.39.0, and point `PYTORCH_LIBRARY_PATH` at a
libtorch 2.7.1+cu128 build from download.pytorch.org (~2.5 GB). That
is the first combination that could execute on this card, and it needs
a machine with the device visible;
3. or bind-mount `/dev/nvidia*` and the host's `libcuda.so.1` into the
sandbox — only the host owner can do that, and it does not help until
step 2 is done.
Until then the CPU leg of `test/test-vchord.sh` and `v-chord-e2e` in the
CWL suite remain the GPU-less proofs.
|