diff options
| -rw-r--r-- | topics/pangenome/impg/impg-agc-bindings.gmi | 76 | ||||
| -rw-r--r-- | topics/rust/guix-rust-bootstrap.gmi | 144 |
2 files changed, 220 insertions, 0 deletions
diff --git a/topics/pangenome/impg/impg-agc-bindings.gmi b/topics/pangenome/impg/impg-agc-bindings.gmi index b190b1d..ef7c55b 100644 --- a/topics/pangenome/impg/impg-agc-bindings.gmi +++ b/topics/pangenome/impg/impg-agc-bindings.gmi @@ -23,6 +23,16 @@ It should be straightforward to create a Rust module that calles into the shared One early choice is a separation of concerns. We will try to build the library independently of the Rust package. This follows a standard model. For example cargo should not build zlib - it is provided by the environment. The bindings, meanwhile, are defined and built in cargo. +# Tasks + +* [ ] Get guix to compile impg with AGC +* [ ] Add optimization +* [ ] Make sure spoa build in spoa-rs is optimized +* [ ] Create static binary for distribution +* [ ] Create singularity example + +# Steps + ## Setting up Guix with rust Guix provides a reproducible build environment. If you get over the fact that it is Lisp, it proves a remarkably nice way to handle dependencies. The first step is to set up guix so you get a recent set of dependencies. For this run guix pull and set it up in a profile @@ -133,3 +143,69 @@ We will also write a * [ ] Guix build that creates an optimized impg And that last one allows us to distribute prebuilt binaries in CONDA and apptainer/singularity/docker. + +Note that this is the same approach as taken by + +=> https://github.com/rust-lang/libz-sys/blob/main/build.rs + +which binds against libz. It *optionally* builds the source tree of zlib which is included as a submodule + +=> https://github.com/rust-lang/libz-sys/tree/main/src + +In our case, a rebuild can be useful when AGC lib can not be found. Note that the cargo edition of libz-sys does not invoke make or cmake. It builds it by 'hand'! + +There is also libz-rs, but that is a somewhat typical Rust rewrite of libz: + +=> https://github.com/trifectatechfoundation/zlib-rs + +I also took a quick look at the rust spoa crate. Here a build is always forced, but I don't think it actually optimizes the build. Add a note to my tasks. + +## First guix package by Fred + +Fred drafted a first guix package which can build impg with + +``` +guix build -L .guix/modules -f guix.scm + +/gnu/store/cdjiq6aalpc849hl8irmbn8xax9mq2b6-impg-0.3.1/bin/impg +Command-line tool for querying overlaps in PAF files + +Usage: impg <COMMAND> + +Commands: + index Create an IMPG index + lace Lace files together (graphs or VCFs) + partition Partition the alignment + query Query overlaps in the alignment + similarity Compute pairwise similarity between sequences in a region + stats Print alignment statistics + +Options: + -h, --help Print help + -V, --version Print version +``` + +It builds against rust 1.85 and uses the new cargo support in Guix. It does not have to rebuild the cargo packages already in guix. Nice and a good start! + +=> https://github.com/pangenome/impg/blob/f5ebaf8b511ee06bdeb193ef509836c26cd4793a/.guix/modules/impg/impg.scm#L4 + +we'll still need to add AGC, static output and optimizations. + +## Adding a guix package for AGC + +As a first step we build a package for AGC that compiles libagc.a using AVX2: + +=> https://github.com/pjotrp/impg/commit/ed16948cc4145ff933a19ba54c3bc1fe4cec709f + +we used the vendored in source for raduls-inplace and isa-l. Not sure they are really required, but I think it is harmless here. + +## Make sure libagc.a is linked to impg + +To create a rust package for binding libagc it is worth reading: + +=> https://doc.rust-lang.org/cargo/reference/build-scripts.html#a-sys-packages + +* The library crate should link to the native library libfoo. This will often probe the current system for libfoo before resorting to building from source. +* The library crate should provide declarations for types and functions in libfoo, but not higher-level abstractions. + +So we should create an agc-rs crate that provides a high-level interface to the upcoming libagc-sys crate. No wonder these crates proliferate. diff --git a/topics/rust/guix-rust-bootstrap.gmi b/topics/rust/guix-rust-bootstrap.gmi new file mode 100644 index 0000000..636aff2 --- /dev/null +++ b/topics/rust/guix-rust-bootstrap.gmi @@ -0,0 +1,144 @@ +# Guix Rust Bootstrap + +To develop code against rust you often need a recent edition of rust. With Guix this is possible because you don't depend on the underlying linux distribution to provide recent versions of glibc and other libraries. Here we have a recipe that should work anywhere on Linux. + +I succeeded in running the latest Rust on Octopus and building packages with guix. + +To make it work the following steps are required: + +* Update guix with guix-pull if your guix is older than 3 months +* Unset GUIX_PROFILE on some systems +* Set your updated guix profile vars +* Create a container that has all dependencies for rust itself +* Run rustup +* Run cargo with LD_LIBRARY_PATH set to $GUIX_ENVIRONMENT/lib + +# Get Guix updated + +Important is to have a recent version of Guix. This is achieved with 'guix pull' and making sure it works. + + +```sh +guix pull -p ~/opt/guix-pull --url=https://codeberg.org/guix/guix +``` + +it takes a few minutes. Next set the environment + +```sh +unset GUIX_PROFILE +. ~/opt/guix-pull/etc/profile +``` + +This will point the path to a recent guix. You can make sure with + +``` +guix describe + guix 772c456 + repository URL: https://codeberg.org/guix/guix + branch: master + commit: 772c456717e755829397a6ff6dba4c1e135426d8 +``` + +which can be validated against the Guix tree. Running + + +```sh +guix package -A rust +rust 1.85.1 rust-src,tools,out,cargo gnu/packages/rust.scm:1454:4 +``` + +shows the current *stable* version in Guix. Now, of course, we want something more to get rust latest. + +# Update Rust and Cargo to latest (stable) + +The trick is to set up a container with Rust in your git working directory: + +``` +mkdir -p ~/.cargo ~/.rustup # to prevent rebuilds +guix shell --share=$HOME/.cargo --share=$HOME/.rustup -C -N -D -F -v 3 guix gcc-toolchain make libdeflate pkg-config xz coreutils sed zstd zlib nss-certs openssl curl +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +. ~/.cargo/env +rustup default stable +``` + +Now rustc shows it is recent: + +``` +rustc --version +rustc 1.90.0 (1159e78c4 2025-09-14) +``` + +Next run 'cargo build' with: + +``` +env LD_LIBRARY_PATH=$GUIX_ENVIRONMENT/lib cargo build + Compiling libagc-sys v0.1.0 (/home/wrk/iwrk/opensource/code/pangenome/libagc-sys) + Finished 'dev' profile [unoptimized + debuginfo] target(s) in 0.06s +$ ./target/debug/libagc-sys +./target/debug/libagc-sys: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory +$ env LD_LIBRARY_PATH=$GUIX_ENVIRONMENT/lib ./target/debug/libagc-sys +Hello, world! +``` + +and your source should build and run. Note the libgcc_s.so.1 error. + +## What if you get a libgcc or librt error? + +The problem is that cargo picks up the wrong libgcc: + +``` +$ ls /gnu/store/*/lib/libgcc_s.so.1 +/gnu/store/m2vhzr0dy352cn59sgcklcaykprrr4j6-gcc-14.3.0-lib/lib/libgcc_s.so.1 +/gnu/store/rbs3nrx9z6sfawn3fa8r8z1kffdbnk8q-gcc-toolchain-15.2.0/lib/libgcc_s.so.1 +/gnu/store/v3bq3shn333kh7m6gj3r58l0v7mkn4in-profile/lib/libgcc_s.so.1 +/gnu/store/xm7i1gvi0i9pyndlkv627r08rsw1ny96-gcc-15.2.0-lib/lib/libgcc_s.so.1 +``` + +This is because Guix itself builds on an older libgcc and librt. You need to tell it explicitly what library to load that built your cargo: + +``` +ldd ~/.cargo/bin/cargo + linux-vdso.so.1 (0x00007ffd409b2000) + libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007fd2cf433000) + librt.so.1 => /lib/librt.so.1 (0x00007fd2cf42e000) +``` + +in the container: + +``` +ls -l /lib/libgcc_s.so.1 +lrwxrwxrwx 1 65534 overflow 82 Jan 1 1970 /lib/libgcc_s.so.1 -> /gnu/store/rbs3nrx9z6sfawn3fa8r8z1kffdbnk8q-gcc-toolchain-15.2.0/lib/libgcc_s.so.1 +``` + +which happens to be the one in $GUIX_ENVIRONMENT/lib! So setting the library path solves it. + +The reason that we don't get the automatically resolving libraries that you normally have in guix is that we have updated rust by *hand* using rustup. Guix has no control over this process. + +# spoa-rs on octopus01 + +I just did above to build spoa-rs. Only had to add cmake to the shell packages. + +# Updating the container + +Now your build may fail because you miss a crucial library or tool. This is a feature of guix containers as it makes dependencies explicit. + +Just add them to the guix shell command. Let's say we add zlib + +``` +guix shell --share=$HOME/.cargo --share=$HOME/.rustup -C -N -D -F -v 3 guix gcc-toolchain make libdeflate pkg-config xz coreutils sed zstd zlib nss-certs openssl curl zlib +``` + +# Troubleshooting + +## Collisions + +Guix may complain about collisions. These are mostly naming issues: + +``` +warning: collision encountered: + /gnu/store/nym6kiinrg2mb8z4lwnvfx5my8df9vrs-glibc-for-fhs-2.41/bin/ldd + /gnu/store/rbs3nrx9z6sfawn3fa8r8z1kffdbnk8q-gcc-toolchain-15.2.0/bin/ldd +warning: choosing /gnu/store/nym6kiinrg2mb8z4lwnvfx5my8df9vrs-glibc-for-fhs-2.41/bin/ldd +``` + +it will like one into your environment. You can still use both tools by using the full path and normally ignore the warning. |