about summary refs log tree commit diff

guix-hmf

Guix packages for the Hartwig Medical Foundation WiGiTS tools: every Java module of the hmftools monorepo (AMBER, PURPLE, SAGE, LINX, ...) plus the shared hmf-common library, the Java dependency tree they need, and CWL descriptions of the pipeline for ravanan.

Everything that can reasonably be compiled here is compiled here: the hmftools modules are built from a pinned upstream source snapshot, and the third-party libraries are built from their Maven Central source archives. Only artifacts that are native payloads, or that cannot be built in this environment, are installed as published jars (see Not built from source).

Layout

hmf/packages/hmftools.scm        hmftools modules + maven-* release-name variants
hmf/packages/java-libraries.scm  third-party Java dependencies
test/test-vchord.sh              end-to-end DJL/PyTorch smoke test
cwl/                             CWL tools, workflows and tests (ravanan)

The channel is used directly from a checkout:

guix build -L . java-purple
guix shell -L . java-purple java-amber java-cobalt -- purple --version

or as a (local-file ...) / git channel entry in ~/.config/guix/channels.scm.

Source pin and version tracking

The hmftools monorepo builds as amber 4.4 / purple 4.5 / sage 5.1 / ... but has published no tag for those versions (the newest release tag is still amber-v4.3). %hmf-source therefore pins the master commit those version properties come from:

field value
commit 6531166b6664981725de58703224b6d8b044336e (2026-10-14)
sha256 1nbr2csac8naz5x21wrza4qlhkykzmk6avn977333ysxn3di9h7g

Each package's version is taken from that tree's pom.xml, so the version strings describe what is actually compiled. When the pin moves, bump the versions together with it (they are all read from the same schema-free <module>.version properties).

%hmf-source-4.3 keeps the old amber-v4.3 snapshot for the two modules upstream has since removed (fastq-tools, health-checker); hmftools-module takes #:source for exactly this case.

Packages

hmftools modules

All built from source by hmftools-module, one package per module, each installing share/java/<module>.jar and a bin/<module> launcher:

package version note
java-amber 4.4
java-bam-tools 1.6.2
java-chord 2.1.2
java-cider 1.2
java-cobalt 3.1
java-compar 1.6
java-crest 1.0.0
java-cuppa 2.5
java-esvee 2.1
java-fastq-tools 1.0 built from the 4.3 tree (dropped upstream)
java-gene-utils 1.3
java-health-checker 3.6 built from the 4.3 tree (dropped upstream)
java-hmf-common 4.4 library, no launcher
java-hmf-common-bwa 4.4 library, keeps common.bwa
java-hmf-id-generator 3.1
java-isofox 2.1
java-lilac 2.1
java-linx 2.4
java-neo 1.2
java-orange 5.1
java-orange-datamodel 5.1 library, no launcher
java-paddle 1.1
java-panel-builder 1.2
java-patient-db 3.0 library, no launcher
java-pave 1.9
java-pave-reverse 1.9
java-peach 2.0
java-purple 4.5
java-qsee 1.0
java-redux 2.1
java-sage 5.1
java-sigs 1.2.1
java-sv-tools 1.0
java-taur 1.0
java-teal 1.4
java-v-chord 1.0.1
java-v-cuppa 0.1
java-virus-interpreter 1.7.1
java-wisp 1.4

maven-amber and maven-bam-tools are the same builds under the names upstream publishes (the amber_v<version>.jar / bam-tools_v<version>.jar release artifacts). They used to be those jars, installed verbatim; they are now compiled from source and keep the release artifacts' visible behaviour: the names, the bamtools alias for bam-tools, and the release launcher that lifts -Xm*, -D* and -XX* arguments out of argv into JVM options (#:launcher-style 'release). Plain module launchers are thin exec java -cp ... MAIN "$@" scripts (#:launcher-style 'thin).

Launchers start with an absolute #!<bash-minimal>/bin/bash: /bin/sh does not exist on a Guix System. Modules without an entry point (java-orange-datamodel) pass #:main-class #f and get no launcher.

Third-party dependencies

hmf/packages/java-libraries.scm holds 78 packages:

category count what
compiled here 57 47 via maven-central-source-jar plus this channel's own from-source packages
published jars 14 decided case by case, see Not built from source
DJL native bundles 7 platform jars that are the libtorch binaries
total 78

Two constructors:

  • maven-central-source-jar — the default. Fetches ARTIFACT-VERSION-sources.jar from Maven Central and compiles it with javac on openjdk17, installing the same <artifact>-<version>.jar name the binary package used, so consumers are unaffected.
  • maven-central-jar — copies the published jar verbatim (see the exceptions below).

maven-central-source-jar handles the things Maven normally does:

concern handling
module-info.java deleted; ant has no module path and javac errors on it
multi-release META-INF/versions/** pruned; ant's flat javac would compile duplicate classes
resources (META-INF/services, *.properties) copied into the class tree before jarring
META-INF/maven/<group>/<artifact>/pom.properties synthesised (DJL reads versions back from it at runtime)
generated resources #:extra-resources, e.g. DJL's pytorch-engine.properties
optional integrations #:prune, a list of regexps (Avalon logging, OSGi annotations, c3p0, Lucene full-text, PDFBox pubkey crypto)
JDK level #:jdk openjdk17; several artifacts need instanceof patterns or java.util.concurrent.Flow

Things that bit us and are worth remembering:

  • BouncyCastle split. iText 7.1.5 compiles against the 1.53 line (DEROutputStream was still public); PDFBox 2.0.30 and 3.0.6 need the 1.68 line (ASN1Encodable.encodeTo). Both are packaged (java-bcprov-jdk15on, -1.68 and matching bcpkix); they only meet in PDFBox's unused certificate-encryption path.
  • jOOQ 3.15.5 is pinned to r2dbc-spi 0.9.0.M1, not the GA — that is what jooq-parent declares, and the GA's Row/connection API does not match.
  • jOOQ codegen needs java-jaxb-{api,runtime}, istack, txw2, jakarta-activation and H2 (as the DDLDatabase engine) on a hand-built classpath; java-patient-db globs each dependency's share/java rather than hard-coding jar names.
  • H2's sources jar is incomplete: org/h2/util/JSR310Utils.java is generated at build time upstream. java-h2database fetches the git tag and compiles h2/src/main together with h2/src/java8/src.
  • boxable 1.8.2 uses CSVFormat.Builder.get(), which is newer than commons-csv 1.9 and drags in commons-io ≥ 2.16; only its datatable helper package is pruned, ORANGE builds tables through BaseTable.
  • hmftools resources carry Maven filter placeholders (${project.version}, ${timestamp}); hmftools-module substitutes them at bundle time or VersionInfo throws DateTimeParseException at startup.
  • hmf-common's resources matter: ant-build-system only compiles src/main/java, so java-hmf-common jars src/main/resources itself. Without it RefGenomeCoordinates NPEs and BAM Tools, AMBER, COBALT, PURPLE and SAGE all die with ExceptionInInitializerError.

Not built from source

group packages, and why it stays a published jar
Kotlin toolchain java-kotlin-* (8), java-kotlinx-coroutines-* (2) — sources are .kt; needs a Kotlin compiler bootstrapped from these jars
native payloads java-djl-pytorch-jni, java-gatk-bwamem-jni, java-djl-pytorch-native-* (7) — the jar is the native code (libdjl_torch.so, libtorch, BWA-MEM)
empty sources java-selenium-java — aggregator pom, its sources jar is empty
deep dependency tree java-tablesaw-core — needs fastutil, RoaringBitmap, univocity, icu4j, classgraph, smile-core, jamm
cannot compile on JDK 17 java-postgresql — waffle 1.9 needs the JAAS java.security.acl API that JDK 17 removed, and scram 2.1 is not published on Central; jOOQ only needs its public API at compile time
shaded fat jar java-immutables-value — upstream ships a shade-plugin jar with relocated guava, consumed only as an annotation processor

The Kotlin group is java-kotlin-stdlib, -stdlib-jdk7, -stdlib-jdk8, java-kotlin-reflect, java-kotlin-compiler, -compiler-embeddable, -daemon-embeddable, plus java-kotlinx-coroutines-core and -core-jvm.

CWL workflows (ravanan)

cwl/ describes the WiGiTS pipeline in CWL and runs it with ravanan, taking every tool binary from this channel's packages via a Guix manifest. Stage hand-offs are Directory values, as the modules' -<stage>_dir flags expect; running that shape needs ravanan built from its checkout with the six patches in cwl/README.md (Directory values, array staging, unlisted outputs). See cwl/README.md for the graph and the full test matrix. In short:

export HMF_GUIX_CHANNEL=$PWD
RAVANAN=/path/to/ravanan HMFTOOLS_DIR=/path/to/hmftools \
  ./cwl/tests/run-tests.sh            # add VCHORD=1 for the PyTorch case
case what it proves (all cases pass)
version-<module> (15) the manifest exposes that module: <module> -version runs inside a ravanan job
peach-haplotypes a real PEACH run on the hmftools haplotype fixtures, producing five output TSVs
bam-metrics a real two-step workflow: samtools index, then BAM Tools' metrics
array-stage an InitialWorkDirRequirement listing a File[] input stages every element
dir-handoff one step's Directory output feeds the next step's Directory input
v-chord-e2e (VCHORD=1) libtorch under ravanan: the PyTorch HRD classifier on the TSO500 model

ravanan must be built from its checkout, not from the 0.2.0 release: the release crashes on any tool without an arguments field, and Directory-based workflows need the patches listed in cwl/README.md (branch cwl-directory-and-array-support). Its own unit suites pass here (58/58); the cwltest-based e2e and conformance suites cannot be built because cwltest's dependency python-cheroot fails two SSL assertions on this OpenSSL.

Verification

# every package in the channel
guix build -L . --no-offload --keep-going \
  $(grep -oP '^\(define-public \Kjava-[a-z0-9.-]+' hmf/packages/hmftools.scm) \
  maven-amber maven-bam-tools

# ditto for the libraries
guix build -L . --no-offload --keep-going \
  $(grep -oP '^\(define-public \Kjava-[a-z0-9.-]+' hmf/packages/java-libraries.scm)

Expected: exit 0. Smoke test that every launcher resolves its main class:

for s in /gnu/store/*-java-*/bin/*; do "$s" 2>&1 | head -3; done

Upstream test suite (Maven, outside Guix)

Guix has no Maven. Using a Maven binary and the store's JDK:

export JAVA_HOME=$(guix build openjdk@17:jdk)
export PATH=/path/to/apache-maven/bin:$JAVA_HOME/bin:$PATH
cd hmftools
mvn -B install -DskipTests                          # hmf-common's test-jar
mvn -B -Dmaven.build.cache.enabled=false test        # 3062 tests, 0 failures

install -DskipTests first: amber's tests need hmf-common's test-jar, which only package/install produces. The build-cache extension is enabled upstream, so a second mvn test silently reuses cached results unless -Dmaven.build.cache.enabled=false is passed.

v-chord end-to-end (DJL / libtorch)

./test/test-vchord.sh

Downloads the upstream TSO500 TorchScript model, fabricates a minimal PURPLE output directory, and runs java-v-chord on CPU (and on GPU if nvidia-smi is present). Expected CPU result for the fabricated COLO829T sample:

breast 0.7854  ovarian 0.8577  pancreatic 0.8223  prostate 0.8063  other 0.8236

The scores are meaningless biologically (the circos PNG is a stub), but every stage of the DJL pipeline exercises: libtorch dlopen, TorchScript deserialisation, forward pass, output translation.

The GPU leg needs a DJL runtime with kernels for the installed card, and no DJL release has them for RTX 50-series (Blackwell, sm_120): the newest native DJL publishes is pytorch-native-cu124 2.5.1, i.e. CUDA 12.4, whose libtorch_cuda.so carries sm_35 … sm_90 kernels only (checked against the 2.2.2/cu121 bundle here). Blackwell needs CUDA 12.8+ (PyTorch 2.7+), so the card fails with "no kernel image is available for execution on the device". The plausible route is a newer DJL (pytorch-jni 2.7.1 exists for DJL 0.34–0.39, and every DJL class v-chord uses is unchanged in 0.39) plus a system libtorch 2.7.1+cu128 pointed at by PYTORCH_LIBRARY_PATH — that has to be tried on a host where the GPU device is actually visible.

  • cwl/README.md — CWL/ravanan side of this repo.
  • skills/guix-containers/SKILL.md, skills/guix-packaging/SKILL.md in the ai-root checkout — container/VM and packaging workflows used with this channel.