about summary refs log tree commit diff
AgeCommit message (Collapse)Author
2 daysslurm-container: 30 CPUs, honest memory, and a one-second scheduler HEAD mainPjotr Prins
The node declared CPUs=2, which is not the container's capacity: the container sees all 96 host CPUs (affinity 0-95) and 754 GB of RAM. Measured with a 30-job probe, the old declaration ran 2 at a time; ravanan's LOCO workflow (20 chromosomes x kinship+gwa) took 102 s cold against 14.3 s single-machine. Now: CPUs=30, RealMemory=65536, DefMemPerCPU=1024 (so a job that requests no memory no longer takes the whole node's), SlurmdParameters=config_overrides (the conf, not slurmd's probe, decides the node's size), and a one-second scheduler/backfill cycle so a 40-job scatter starts in one go. Measured after: 30 of 30 plain jobs RUNNING within a second, and ravanan's cold LOCO run down from 102 s to 24.7 s with the same 21 051 rows. Two things learned the hard way, both now in the comments: * takes the *host* path. Passing the container-side spelling (/tmp/ai-root/slurm-share) drops the mount silently, and every job then dies with signal 53 and no stdout/stderr -- which looks like a Slurm bug, not a missing mount. * a node that declares fewer CPUs than it has is a lie that costs 4x, but declaring more than it has does not oversubscribe the hardware: the CPU count is what Slurm *schedules* on, and the kernel still runs the jobs.
4 daysNote the topology: slurm stays a separate box from the GN4AI webserverPjotr
Comments only; the gn4ai container derivation is byte-identical before and after (bfmdmdznzs2p0wa65z31zpcwzmq0wdk4-run-container.drv), which is the check that nothing but prose moved. gn4ai.scm's header now says why a batch scheduler is not in it: the controller, the compute daemon and munge would join a public webserver's failure domain; slurm, slurmrestd and munge accounts would enter a container that has none; and every slurm configuration change would become a webserver rebuild. GN4 needs the client half -- ravanan and ccwl -- and a --slurm-api-endpoint, which is the production shape anyway: the cluster is another machine, not a neighbour process. It also records the argument in the other direction, since it is real: run-guix-container supervises one container at a time, so the two cannot be up together on one host -- which is a reason to fix the daemon or use a second box, not to merge them. specials/slurm-container.scm says the same from its own side, and that it deploys on a machine of its own as it stands.
4 daysslurm-container: tight scheduler cycles, so a scattered workflow is not ↵Pjotr Prins
waiting on backfill Slurm's default scheduler and backfill cadence is tens of seconds: a job that is ready to run still waits for the next cycle. For a ravanan workflow, which submits its steps a few seconds apart, that wait dominates the run -- six one-second jobs took about three minutes. sched_interval=5, sched_min_interval=1, bf_interval=5 and bf_max_job_start=40 cut it to the job's own cost, which is what makes the 20-chromosome LOCO mapping finish in ~4.5 minutes rather than tens. For a throwaway test container, not a deployment: a real cluster's cadence is a shared policy.
4 daysgn4ai: the container shepherd, from the other containers' fixPjotr
gn4ai.scm had the sysctl/udev no-op workaround but not the fix the rest of the family uses: essential-services with shepherd-for-containers ((genenetwork packages)), the shepherd 1.0.9 that is built with SHEPHERD_DISABLE_SIGNALFD forced on, i.e. sigaction and a self-pipe instead of signalfd. Why it matters, and why it was easy to miss here: as PID 1 in the container daemon's sandbox the signalfd wake-up never fires, so SIGCHLD is reaped but the process monitor never learns, and any service start that waits for a child's exit status -- spawn-command, system*, invoke, the stock nginx's `nginx -t` one-shot -- blocks forever in "starting". gn4ai's own start is a make-forkexec-constructor, which is unaffected, so the container booted and the omission sat there waiting for the next one-shot anyone adds. Same shape as cgit-container.scm, genecup-container.scm, julia-notebook-container.scm, gn-guile-webserver-container.scm, jbrowse2-container.scm and fallback.scm. Verified after the rebuild: shepherd resolves to the self-pipe build, `herd status` lists the services, :3003 answers /version in 2 s, and /compute/gemma (21 055 rows), /compute/qtl2 (20 453 rows) and /compute/bnw (30 edges) all return 200.
4 daysgn4ai: GEMMA_DIR for the engine, and run from the code treePjotr
The wrapper now takes --gemma-dir/GEMMA_DIR, so the container stops pretending that a PATH is a configuration: the service sets GEMMA_DIR to the engine profile on the shared tree, and gn4.env names the same directory once (GEMMA_WRAPPER_CMD stays, because that is what GN4 reads). PATH keeps the directory only for pfff, which the wrapper calls by name. The service also cd's into the code tree before exec, as production does, and gn4.env names the R and python scripts by absolute path (QTL2_SCRIPT_PATH, GEMMA_PLOTTER, BULKLMM_SCRIPT, BNW_PLOTTER). Both were needed: /compute/qtl2 answered qtl2_failed Fatal error: cannot open file 'scripts/qtl2_scan.R' in the container, because a shepherd service's working directory is not the source tree -- a path in a config file should not depend on anyone's cwd. Verified against the running container: /compute/gemma 200 (21 055 rows), /compute/qtl2 200 (20 453 rows), /compute/bnw 200 (30 edges), /plot/bnw 200 with an SVG, /version in 2 s.
4 daysgn4ai: gemma-wrapper in a directory of its ownPjotr
gemma-wrapper was in the container's package list, which is wrong for the same reason a store path would be: it is an engine, not a tool for whoever ssh-es in, and the container profile then carries a version that can drift from the one the API actually runs. It now lives in a Guix profile of its own, on the shared gn4-container tree (gn4-container/gemma-wrapper), and gn4.env names that directory: GEMMA_WRAPPER_DIR .../gn4-container/gemma-wrapper/bin GEMMA_WRAPPER_CMD $GEMMA_WRAPPER_DIR/gemma-wrapper PATH $GEMMA_WRAPPER_DIR:/run/current-system/profile/bin:... A directory rather than one store path, because this is measured: with only the wrapper pinned, /compute/gemma answered 502 twice, first with Errno::ENOENT inside Open3.popen_run for `time -v gemma ...` and then, after GNU time was added, with "env: 'tar': No such file or directory". The wrapper shells out to its neighbours, so gemma, time, tar, parallel and xz are installed beside it. The profile is on the share (so it outlives the container) and pins its own closure (so a guix gc cannot take the engine away). The container definition uses the directory too -- the service sets GEMMA_WRAPPER_DIR and PATH from it -- so a half-written env file cannot leave the engine unfindable. With gemma-wrapper out of the package list, nothing in this file imports guix-bioinformatics any more; -L is needed only for (genenetwork services container-ssh). Verified in the running container: /compute/gemma 200 with 21 055 rows, /compute/bnw 200 with 30 edges, /groups/BXD 200, /version in 2 s, and gemma-wrapper absent from the system profile while the service still finds it.
4 daysgn4ai.scm: a GN4AI container, MariaDB and gn-auth outside itPjotr
A store-sharing container for genenetwork4, named where the other machine definitions are (the repo root, as asked): service on :3003, ssh on :2223, because 3000/3001 are the host's own instances, 8081 the gn-auth stub, 2222 another container's ssh and 3306 MariaDB. MariaDB and gn-auth stay outside. --network shares the host's network namespace, so 127.0.0.1 inside is the host's loopback and SQL_URI keeps resolving; nothing here reaches into either. Paths, because this is where containers actually break: --expose (ro) gn4-data (42 GB), gn4-index (97 GB), the code tree (the R/python scripts the API runs, the gn-bnw driver), ~/.julia --share (rw) factory/gn4-container -- gn4.env (0600: SQL_URI's password, every directory, the ports), HOME, TMPDIR, tokens.sqlite, baskets, the Wikidata cache and every engine cache, since a container's root is ephemeral gn4.env is a file rather than fields in this .scm because SQL_URI carries a password and store paths are world-readable. HOME is in there for the reason the slurm container taught: gemma-wrapper locks $HOME/.<hash>.lck and dies without a writable one. Four things that only failed at run time, each fixed here: * a bare shepherd-service cannot go to (service ...) -- that wants a service-type -- so it is (simple-service 'gn4ai shepherd-root-service-type (list gn4ai-service)); * (container-ssh-service ...) returns a LIST of services, so it is appended, not wrapped in (list ...); * %gn4-binary must be the STORE path, not the GC-root symlink under factory/: only the store and the listed paths are mounted inside, so the symlink does not exist there (it fails as a 117-byte log line saying nothing); * the package list is the service's *runtime*, not decoration: without python /compute/bnw answered "python3: No such file or directory" and without gemma-wrapper /compute/gemma answered "failed to run gemma-wrapper". Both were measured before adding python, gemma-wrapper, r and the r-* packages the scan scripts load. Verified against the running container: /version in 2 s, /groups/BXD (200, database reachable through the shared netns), /compute/bnw (200, 30 edges), /compute/gemma (200, 21 055 rows), and ssh on :2223 showing the gn4ai service started. Also adds the ai-memory store for this repo.
4 dayscgit: Move the /plain/ exemption into the Anubis policy.Pjotr
The nginx-side bypass added for raw downloads answered 403 to any user agent outside the allowlist, which also broke cgit's "plain" link for browsers, and it put bot policy in two places (an nginx location plus the Anubis policy). Serve /plain/ through Anubis like everything else and let the policy allow it: %anubis-challenge-all-policy now has a `plain-download-tools' rule (CEL, since user_agent_regex and path_regex cannot be combined) that ALLOWs curl/wget on /<repo>/plain/ when the request carries no Sec-Fetch-* header. Tools fetch raw files and cannot solve a proof-of-work challenge; browsers do send Sec-Fetch-*, so they still get the challenge -- and can solve it, so the plain link works again. A browser claiming a curl user agent is challenged too. Header names in the expression must be in canonical Go form (Sec-Fetch-Mode): the X-Forwarded-For style lookup is case-sensitive. Drops the plain-anubis-bypass? and plain-user-agents fields, the cgit-plain-location, and the now unused nginx-map serializer. Tests t23-t26 cover the policy: curl/wget are served, browsers and spoofed curl user agents are challenged (the latter two are Anubis-only, like t17/t18).
4 daysanubis: Honour the configuration other services contribute.Pjotr
fold-services only applies extensions when a service type defines `extend' -- without it the instance keeps its own value and the contributed values are dropped. anubis-service-type defined neither compose nor extend, so cgit-nginx-service-type's contribution (the Anubis target, policy and difficulty derived from the cgit configuration) never took effect: Anubis always ran with the default configuration. It went unnoticed because the default target happened to be the internal listener address. Compose the contributions and prefer a contributed configuration over the default one, and export the target accessor so the extension can set the field.
7 daysgenenetwork2: Upgrade to commit 604d5ea8.Frederick Muriuki Muriithi
7 daysgenenetwork3: Upgrade to commit a090c15.Frederick Muriuki Muriithi
7 daysgenenetwork3: Update the source uri.Frederick Muriuki Muriithi
14 daysgenenetwork2: Upgrade to commit bc4f8b4.Frederick Muriuki Muriithi
14 daysgn-libs: Upgrade to commit cf930d1.Frederick Muriuki Muriithi
2026-10-28genenetwork2: Upgrade to commit 5ce0d93.Frederick Muriuki Muriithi
2026-10-28genenetwork2: Upgrade to commit 2ce6efa.Frederick Muriuki Muriithi
2026-10-28Partially revert "Started to update the fallback container"Frederick Muriuki Muriithi
This reverts parts of commit 800bf303626f29b376a529aae7beff11c24eec51, specifically where `rust-qtlreaper` is sourced from. In the newer guix-bioinformatics, `rust-qtlreaper` is sourced from (gn packages genenetwork-rust) but a change to use that package will force a migration to use a newer guix version with the attendant cascade of package breakages due to an upgrade. This commit reverts to sourcing `rust-qtlreaper` from (gn packages genenetwork) as was previously done in the older guix-bioinformatis commits.
2026-10-28gn-libs: Upgrade to commit 6c67a0a.Frederick Muriuki Muriithi
2026-10-28cgit: Test per-client budgets and /plain/ handling.Pjotr
The rate-limit tests could not catch the shared-bucket bug: they always spoke from one address, so "per client IP" and "per proxy IP" looked identical. Give each test its own X-Forwarded-For bucket (the listener trusts 127.0.0.1, see trusted-proxies) and add the regression: * t19/t20 keep covering the expensive and general limits, now with the 60r/m burst 20 and a bucket nobody else spends; * t21 exhausting one forwarded client's expensive budget must leave the next one alone; * t22 the same for the general (server-level) budget; * t23/t24 /plain/ serves curl and wget and answers 403 to anything else; * t25 /plain/ is not challenged by Anubis. t10 identifies as curl now, because /plain/ is an allowlist.
2026-10-28cgit: Rate limit per forwarded client and serve /plain/ without Anubis.Pjotr
Every limit was keyed on $binary_remote_addr, but the deployment terminates TLS on the host and forwards from 127.0.0.1 without X-Forwarded-For, so the four limits collapsed into one bucket shared by the whole internet: 5 r/s, 20 connections and 60 r/min for /log|/commit|/diff for everybody. Ordinary browsing got 429s (9 of 12 requests at one request every two seconds), and `git clone' failed on its first request. Recover the client address instead: new trusted-proxies and real-ip-header fields feed set_real_ip_from/real_ip_header/ real_ip_recursive, emitted by the new cgit-http-directives (the OS passes it as nginx extra-content; %cgit-rate-limit-http-directives is kept as the default-config value). The trusted-proxies default of 127.0.0.1 matches the host proxy, and set to '() when an instance is exposed directly so a client cannot pick its own bucket. Also for anonymous reads: * the expensive zone goes from 5r/m burst 3 to 60r/m burst 20 -- one request per second per client still stops history walks but leaves log -> commit -> diff chains alone; * /plain/ (raw file content: curl, wget, scripts) can skip the Anubis challenge (plain-anubis-bypass?), which a tool cannot solve, and can be restricted to user-agent prefixes (plain-user-agents) so it cannot be used from a browser; * internal-listen makes the Anubis-mode internal cgit address configurable, since a --network container shares the host netns and port 8081 may be taken. The container uses nginx-realip (guix-bioinformatics, gn packages web): the stock nginx is built without the real_ip module and refuses the directives with `unknown directive "set_real_ip_from"'.
2026-10-28nginx: Add real_ip and map serializers.Pjotr
set_real_ip_from / real_ip_header / real_ip_recursive let nginx recover the client address from X-Forwarded-For when it runs behind a reverse proxy, so directives keyed on $binary_remote_addr stay per-client instead of per-proxy. nginx-map emits a map block, used for the user-agent allowlists that need a variable rather than a hand-built string.
2026-10-28Started to update the fallback containerPjotr Prins
2026-10-28SlurmPjotr Prins
2026-10-28JBrowse2 containerPjotr Prins
2026-10-28slurm-container: AuthAltTypes for the REST plane, munge for the daemons.Pjotr
Job execution under slurm-api works now. The cluster-wide AuthType=auth/jwt was the bug: auth_jwt only loads the verification key in slurmctld/slurmdbd, so slurmd could not decode the credential and every slurmctld -> slurmd launch RPC arrived as SLURM_AUTH_NOBODY (uid 99, "Security violation") -- which is why submissions were recorded but never ran. Wire it the way Slurm expects instead: - AuthType=auth/munge for the daemon plane (slurmd verifies by uid), - AuthAltTypes=auth/jwt so slurmctld alone also accepts JWT-signed REST RPCs (the plugin loader reads AuthAltTypes only in slurmctld/slurmdbd), - SLURM_JWT in slurmrestd's environment only, which makes auth_g_init() pick auth/jwt for that process so rest_auth/jwt's per-request identity switch (always on the primary plugin) is accepted; the caller's token is forwarded and slurmctld decodes it to the real uid. wait-for-munge-then-exec grew a positional JWT? flag (Guile's define* does not take a rest arg after #:key), so only slurmrestd gets the token. Also link /bin/bash and /bin/sh at activation: slurm runs a batch script through its shebang and ravanan submits "#!/bin/bash" + the store script, which a minimal Guix System cannot resolve. Verified: ravanan's slurm-api batch system submits and completes real slurm jobs in this container, for a single-step and a two-step workflow. The session notes (slurm-session.md, kept out of git by a *session.md ignore rule) document the resolution and the repro.
2026-10-27gn-libs: Upgrade to commit 366237f.Frederick Muriuki Muriithi
2026-10-27gn-uploader: Upgrade to commit 2bc7e2c.Frederick Muriuki Muriithi
2026-10-27gn-auth: Upgrade to commit a50a966.Frederick Muriuki Muriithi
2026-10-26genenetwork2: Upgrade to commit 3b39746.Frederick Muriuki Muriithi
2026-10-22Update gn-uploader WSGI module pathpi coding agent
gn-uploader's `scripts` package moved to `uploader/scripts` (now `uploader.scripts`), so the gunicorn app module for the gn-uploader service is now `uploader.scripts.qcapp_wsgi:app`.
2026-10-22slurm container (WIP)Pjotr Prins
2026-10-22cgit: port 8011Pjotr Prins
2026-10-19README: debugging a container started by run-guix-containerPjotr
Document the GeneCup container as the worked example for the containers built with 'guix system container' and started through the run-guix-system-container daemon, since that is where the failures have actually been found and where nothing was written down: - the three -L checkouts and the two host layouts, plus the trap that a wrong -L reports "genecup-service-type: unbound variable" and nothing else, with the guix repl one-liner that shows the real error; - --expose of the Gemini key and why the activation copies it (owner and 0400 mode that server.py insists on, secret never in the store); - starting through the daemon, boot log via --log, and ssh access; - the log map: /var/log/genecup.log (app, including the per-attempt API errors and traceback), /var/log/gemini-check.log (boot check calling the model), /var/log/messages, the credential file's owner/mode; - the buffering trap (PYTHONUNBUFFERED) and that only successful classifications are cached; - what "Gemini classification not available" now says in each of its three cases, and a curl one-liner that tests the model with the key; - the container test script and its --expect-missing mode.
2026-10-19genecup: run the Flask app unbufferedPjotr
Shepherd redirects the app's stdout to #:log-file ("/var/log/genecup.log"), so Python block-buffers it and the diagnostics that explain a failed request ("Gemini batch attempt 1/3 failed: ...", "Gemini response: ...") can trail the page by kilobytes -- the log looks empty exactly when it is needed. Set PYTHONUNBUFFERED=1 in the service environment.
2026-10-19genecup: boot check calls the model, not just the models cataloguePjotr
A key can list gemini-3-flash-preview in /v1beta/models and still not be allowed to call it (preview gating, project/quota/region limits). In that case the app's stress view prints "All related sentences (Gemini classification not available or no specific stress types found)" after three failed attempts, which reads like "nothing to classify" and hides the API's own error -- exactly the report from a second deployment where the very same key/container works elsewhere. Make gemini-check do what the app does: POST /v1beta/models/<model>:generateContent with a one-sentence classification prompt (a handful of tokens), and on failure quote the API's response body so the boot log names the reason (model not found, permission denied, quota, TLS, ...). The model is a single definition (%gemini-model, gemini-3-flash-preview) that must stay in sync with server.py's hard-coded batch classifier, and the docstring says so. Verdict is now Gemini classification available: key in PATH works, MODEL answered (HTTP 200) Gemini classification not available: MODEL call failed (HTTP 404): {...} so the container tests assert "answered (HTTP 200)".
2026-10-19genecup: expose the Gemini API key and check it at bootPjotr
The app reads its key from ~/.config/gemini/credentials of the genecup user (i.e. <datadir>/.config/gemini/credentials) and exits when the mode is not 0400; when the file is missing it warns and keeps running without LLM features. Nothing brought the host's key into the container, and nothing said whether a key that was there actually worked before a user hit the Gemini path in the UI. New genecup-configuration field 'gemini-credentials': the path where the launcher exposes the host key read-only (a secret, so never in the store). The activation copies it to <datadir>/.config/gemini/credentials with owner genecup and mode 0400 -- a bind mount would keep the host owner, which the genecup user cannot read -- and complains when it is not exposed. New one-shot shepherd service 'gemini-check' reports at boot, in the app's own wording, whether the key exists, has mode 0400, is non-empty and is *accepted by the API* (GET /v1beta/models costs no tokens: 200 usable, 400/403 rejected). Verdict goes to the boot log and /var/log/gemini-check.log. It is a program-file rather than a start gexp because use-modules inside a shepherd start value does not take effect (it compiles to an unbound-variable failure at runtime), and it sets SSL_CERT_FILE since bare curl in a minimal container fails verification with "curl: (60) ... (CAfile: none)". genecup-container.scm exposes the host key and documents the flag; it also documents the -L layouts and the trap found on a second host: a wrong -L makes (forge anubis) unresolvable, so (genenetwork services genecup) compiles to nothing and guix only reports "genecup-service-type: unbound variable". New test genenetwork/services/test/run_genecup_container_tests.sh builds with the key exposed, starts through the daemon and asserts the file owner/mode, the check verdict, GEMINI_API_KEY=set in the app log and the app serving on 4202; --expect-missing inverts it for the unkeyed container. Checkout locations are auto-detected, so it also runs on flat layouts (balg01 ~/services/genecup-service). Both modes pass here.
2026-10-19julia-notebook: drop JULIA_PKG_OFFLINE for the modern stackPjotr
The modern pluto-notebook installs its Pluto/PlutoSliderServer environment with Pkg on first run, so Pkg must be online (the nss-certs bundle already lets it reach the registry over HTTPS). A mounted/persistent depot makes subsequent runs offline.
2026-10-15Remove 'gn-machines/guix' load-path from invocation.Frederick Muriuki Muriithi
The 'gn-machines' repo is defined as a channel in its own right, with the `(directory "guix")` explicitly specifying which part of the repository to use for the channel. Specifying the `--load-path=./guix/` again on the CLI invocation short-circuits that and ensures that you can only ever use the most current local checkout -- preventing the use of different profiles (and profile generations) for building the containers.
2026-10-15genenetwork2: Upgrade to commit 63c809b.Frederick Muriuki Muriithi
2026-10-15genenetwork3: Upgrade to commit ddd5d2d.Frederick Muriuki Muriithi
2026-10-15gn-guile: Upgrade to commit 167eee5.Frederick Muriuki Muriithi
2026-10-15Update phases for build to more easily see progress.Frederick Muriuki Muriithi
2026-10-15gn-auth: Upgrade to commit e7b2a13.Frederick Muriuki Muriithi
2026-10-14julia-notebook tests: retry while the slider server restartsPjotr
PlutoSliderServer on this stack can abort and be restarted by the container's wrapper, so the server is briefly unavailable in cycles. Make the HTTP client retry connection errors (and poll) until the server answers, so the tests are robust to that.
2026-10-14julia-notebook: run Pkg offline; add the container testsPjotr
Set JULIA_PKG_OFFLINE=true for the notebook server: the packaged dependencies are already on the read-only JULIA_LOAD_PATH, and letting Pkg hit the registry crashed the server mid-run (a libcurl "curlmultiassign" error). Notebooks whose packages are not packaged now fail cleanly instead of taking the server down. Add julia_notebook_http_tests.py -- the index is served, the notebooks are listed, and the power notebook runs and is served as power.html -- and run_julia_notebook_container_tests.sh to drive the build/start/test/log-harvest cycle.
2026-10-13julia-notebook: add the Pluto notebook container and servicePjotr
genenetwork/services/julia-notebook.scm: a julia-notebook service-type running pluto-notebook's runsliderserver as an unprivileged 'julia' user, with a configurable port, notebooks directory and Julia depot (JULIA_DEPOT_PATH, a single dir or a colon-separated stack), and the nss-certs CA bundle (SSL_CERT_FILE/CURL_CA_BUNDLE) so Julia's HTTP stack can reach the package registry. genenetwork/containers/julia-notebook-container.scm: the system container around that service -- PlutoSliderServer on port 8890, ssh on 2225 -- following the other containers' conventions (container-ssh, no-op sysctl/udev, no guix service). Host mappings (the Julia depot, real notebooks) are documented as --share/--expose at build time. The copied specials/pluto.scm it supersedes is removed.
2026-10-07gn-auth: Upgrade to commit 02ffa57.Frederick Muriuki Muriithi
2026-10-07gn-auth: Upgrade to commit 0f75d18.Frederick Muriuki Muriithi
2026-10-07genenetwork2: Upgrade to commit 61fc1c2.Frederick Muriuki Muriithi
2026-10-07genenetwork3: Upgrade to commit 0686b8b.Frederick Muriuki Muriithi