| Age | Commit message (Collapse) | Author |
|
The node declared CPUs=2, which is not the container's capacity: the
container sees all 96 host CPUs (affinity 0-95) and 754 GB of RAM. Measured
with a 30-job probe, the old declaration ran 2 at a time; ravanan's LOCO
workflow (20 chromosomes x kinship+gwa) took 102 s cold against 14.3 s
single-machine.
Now: CPUs=30, RealMemory=65536, DefMemPerCPU=1024 (so a job that requests no
memory no longer takes the whole node's), SlurmdParameters=config_overrides
(the conf, not slurmd's probe, decides the node's size), and a one-second
scheduler/backfill cycle so a 40-job scatter starts in one go.
Measured after: 30 of 30 plain jobs RUNNING within a second, and ravanan's
cold LOCO run down from 102 s to 24.7 s with the same 21 051 rows.
Two things learned the hard way, both now in the comments:
* takes the *host* path. Passing the
container-side spelling (/tmp/ai-root/slurm-share) drops the mount
silently, and every job then dies with signal 53 and no stdout/stderr --
which looks like a Slurm bug, not a missing mount.
* a node that declares fewer CPUs than it has is a lie that costs 4x, but
declaring more than it has does not oversubscribe the hardware: the CPU
count is what Slurm *schedules* on, and the kernel still runs the jobs.
|
|
Comments only; the gn4ai container derivation is byte-identical before and
after (bfmdmdznzs2p0wa65z31zpcwzmq0wdk4-run-container.drv), which is the
check that nothing but prose moved.
gn4ai.scm's header now says why a batch scheduler is not in it: the
controller, the compute daemon and munge would join a public webserver's
failure domain; slurm, slurmrestd and munge accounts would enter a
container that has none; and every slurm configuration change would become
a webserver rebuild. GN4 needs the client half -- ravanan and ccwl -- and a
--slurm-api-endpoint, which is the production shape anyway: the cluster is
another machine, not a neighbour process. It also records the argument in
the other direction, since it is real: run-guix-container supervises one
container at a time, so the two cannot be up together on one host -- which
is a reason to fix the daemon or use a second box, not to merge them.
specials/slurm-container.scm says the same from its own side, and that it
deploys on a machine of its own as it stands.
|
|
waiting on backfill
Slurm's default scheduler and backfill cadence is tens of seconds: a job
that is ready to run still waits for the next cycle. For a ravanan
workflow, which submits its steps a few seconds apart, that wait
dominates the run -- six one-second jobs took about three minutes.
sched_interval=5, sched_min_interval=1, bf_interval=5 and
bf_max_job_start=40 cut it to the job's own cost, which is what makes the
20-chromosome LOCO mapping finish in ~4.5 minutes rather than tens.
For a throwaway test container, not a deployment: a real cluster's cadence
is a shared policy.
|
|
gn4ai.scm had the sysctl/udev no-op workaround but not the fix the rest
of the family uses: essential-services with shepherd-for-containers
((genenetwork packages)), the shepherd 1.0.9 that is built with
SHEPHERD_DISABLE_SIGNALFD forced on, i.e. sigaction and a self-pipe
instead of signalfd.
Why it matters, and why it was easy to miss here: as PID 1 in the
container daemon's sandbox the signalfd wake-up never fires, so SIGCHLD is
reaped but the process monitor never learns, and any service start that
waits for a child's exit status -- spawn-command, system*, invoke, the
stock nginx's `nginx -t` one-shot -- blocks forever in "starting".
gn4ai's own start is a make-forkexec-constructor, which is unaffected, so
the container booted and the omission sat there waiting for the next
one-shot anyone adds.
Same shape as cgit-container.scm, genecup-container.scm,
julia-notebook-container.scm, gn-guile-webserver-container.scm,
jbrowse2-container.scm and fallback.scm. Verified after the rebuild:
shepherd resolves to the self-pipe build, `herd status` lists the
services, :3003 answers /version in 2 s, and /compute/gemma (21 055
rows), /compute/qtl2 (20 453 rows) and /compute/bnw (30 edges) all
return 200.
|
|
The wrapper now takes --gemma-dir/GEMMA_DIR, so the container stops
pretending that a PATH is a configuration: the service sets GEMMA_DIR to
the engine profile on the shared tree, and gn4.env names the same directory
once (GEMMA_WRAPPER_CMD stays, because that is what GN4 reads). PATH keeps
the directory only for pfff, which the wrapper calls by name.
The service also cd's into the code tree before exec, as production does,
and gn4.env names the R and python scripts by absolute path
(QTL2_SCRIPT_PATH, GEMMA_PLOTTER, BULKLMM_SCRIPT, BNW_PLOTTER). Both were
needed: /compute/qtl2 answered
qtl2_failed Fatal error: cannot open file 'scripts/qtl2_scan.R'
in the container, because a shepherd service's working directory is not the
source tree -- a path in a config file should not depend on anyone's cwd.
Verified against the running container: /compute/gemma 200 (21 055 rows),
/compute/qtl2 200 (20 453 rows), /compute/bnw 200 (30 edges), /plot/bnw 200
with an SVG, /version in 2 s.
|
|
gemma-wrapper was in the container's package list, which is wrong for the
same reason a store path would be: it is an engine, not a tool for
whoever ssh-es in, and the container profile then carries a version that
can drift from the one the API actually runs.
It now lives in a Guix profile of its own, on the shared gn4-container
tree (gn4-container/gemma-wrapper), and gn4.env names that directory:
GEMMA_WRAPPER_DIR .../gn4-container/gemma-wrapper/bin
GEMMA_WRAPPER_CMD $GEMMA_WRAPPER_DIR/gemma-wrapper
PATH $GEMMA_WRAPPER_DIR:/run/current-system/profile/bin:...
A directory rather than one store path, because this is measured: with
only the wrapper pinned, /compute/gemma answered 502 twice, first with
Errno::ENOENT inside Open3.popen_run for `time -v gemma ...` and then,
after GNU time was added, with "env: 'tar': No such file or directory".
The wrapper shells out to its neighbours, so gemma, time, tar, parallel
and xz are installed beside it. The profile is on the share (so it
outlives the container) and pins its own closure (so a guix gc cannot
take the engine away).
The container definition uses the directory too -- the service sets
GEMMA_WRAPPER_DIR and PATH from it -- so a half-written env file cannot
leave the engine unfindable. With gemma-wrapper out of the package list,
nothing in this file imports guix-bioinformatics any more; -L is needed
only for (genenetwork services container-ssh).
Verified in the running container: /compute/gemma 200 with 21 055 rows,
/compute/bnw 200 with 30 edges, /groups/BXD 200, /version in 2 s, and
gemma-wrapper absent from the system profile while the service still
finds it.
|
|
A store-sharing container for genenetwork4, named where the other
machine definitions are (the repo root, as asked): service on :3003, ssh
on :2223, because 3000/3001 are the host's own instances, 8081 the
gn-auth stub, 2222 another container's ssh and 3306 MariaDB.
MariaDB and gn-auth stay outside. --network shares the host's network
namespace, so 127.0.0.1 inside is the host's loopback and SQL_URI keeps
resolving; nothing here reaches into either.
Paths, because this is where containers actually break:
--expose (ro) gn4-data (42 GB), gn4-index (97 GB), the code tree (the
R/python scripts the API runs, the gn-bnw driver),
~/.julia
--share (rw) factory/gn4-container -- gn4.env (0600: SQL_URI's
password, every directory, the ports), HOME,
TMPDIR, tokens.sqlite, baskets, the Wikidata cache and
every engine cache, since a container's root is
ephemeral
gn4.env is a file rather than fields in this .scm because SQL_URI carries
a password and store paths are world-readable. HOME is in there for the
reason the slurm container taught: gemma-wrapper locks $HOME/.<hash>.lck
and dies without a writable one.
Four things that only failed at run time, each fixed here:
* a bare shepherd-service cannot go to (service ...) -- that wants a
service-type -- so it is (simple-service 'gn4ai shepherd-root-service-type
(list gn4ai-service));
* (container-ssh-service ...) returns a LIST of services, so it is
appended, not wrapped in (list ...);
* %gn4-binary must be the STORE path, not the GC-root symlink under
factory/: only the store and the listed paths are mounted inside, so
the symlink does not exist there (it fails as a 117-byte log line
saying nothing);
* the package list is the service's *runtime*, not decoration: without
python /compute/bnw answered "python3: No such file or directory" and
without gemma-wrapper /compute/gemma answered "failed to run
gemma-wrapper". Both were measured before adding python,
gemma-wrapper, r and the r-* packages the scan scripts load.
Verified against the running container: /version in 2 s, /groups/BXD
(200, database reachable through the shared netns), /compute/bnw (200, 30
edges), /compute/gemma (200, 21 055 rows), and ssh on :2223 showing the
gn4ai service started. Also adds the ai-memory store for this repo.
|
|
The nginx-side bypass added for raw downloads answered 403 to any user
agent outside the allowlist, which also broke cgit's "plain" link for
browsers, and it put bot policy in two places (an nginx location plus
the Anubis policy).
Serve /plain/ through Anubis like everything else and let the policy
allow it: %anubis-challenge-all-policy now has a `plain-download-tools'
rule (CEL, since user_agent_regex and path_regex cannot be combined)
that ALLOWs curl/wget on /<repo>/plain/ when the request carries no
Sec-Fetch-* header. Tools fetch raw files and cannot solve a
proof-of-work challenge; browsers do send Sec-Fetch-*, so they still get
the challenge -- and can solve it, so the plain link works again. A
browser claiming a curl user agent is challenged too.
Header names in the expression must be in canonical Go form
(Sec-Fetch-Mode): the X-Forwarded-For style lookup is case-sensitive.
Drops the plain-anubis-bypass? and plain-user-agents fields, the
cgit-plain-location, and the now unused nginx-map serializer. Tests
t23-t26 cover the policy: curl/wget are served, browsers and spoofed
curl user agents are challenged (the latter two are Anubis-only, like
t17/t18).
|
|
fold-services only applies extensions when a service type defines
`extend' -- without it the instance keeps its own value and the
contributed values are dropped. anubis-service-type defined neither
compose nor extend, so cgit-nginx-service-type's contribution (the
Anubis target, policy and difficulty derived from the cgit
configuration) never took effect: Anubis always ran with the default
configuration. It went unnoticed because the default target happened to
be the internal listener address.
Compose the contributions and prefer a contributed configuration over
the default one, and export the target accessor so the extension can set
the field.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This reverts parts of commit 800bf303626f29b376a529aae7beff11c24eec51,
specifically where `rust-qtlreaper` is sourced from. In the newer
guix-bioinformatics, `rust-qtlreaper` is sourced from (gn packages
genenetwork-rust) but a change to use that package will force a migration to
use a newer guix version with the attendant cascade of package breakages due
to an upgrade.
This commit reverts to sourcing `rust-qtlreaper` from (gn packages
genenetwork) as was previously done in the older guix-bioinformatis commits.
|
|
|
|
The rate-limit tests could not catch the shared-bucket bug: they always
spoke from one address, so "per client IP" and "per proxy IP" looked
identical. Give each test its own X-Forwarded-For bucket (the listener
trusts 127.0.0.1, see trusted-proxies) and add the regression:
* t19/t20 keep covering the expensive and general limits, now with the
60r/m burst 20 and a bucket nobody else spends;
* t21 exhausting one forwarded client's expensive budget must leave the
next one alone;
* t22 the same for the general (server-level) budget;
* t23/t24 /plain/ serves curl and wget and answers 403 to anything else;
* t25 /plain/ is not challenged by Anubis.
t10 identifies as curl now, because /plain/ is an allowlist.
|
|
Every limit was keyed on $binary_remote_addr, but the deployment
terminates TLS on the host and forwards from 127.0.0.1 without
X-Forwarded-For, so the four limits collapsed into one bucket shared by
the whole internet: 5 r/s, 20 connections and 60 r/min for
/log|/commit|/diff for everybody. Ordinary browsing got 429s (9 of 12
requests at one request every two seconds), and `git clone' failed on
its first request.
Recover the client address instead: new trusted-proxies and
real-ip-header fields feed set_real_ip_from/real_ip_header/
real_ip_recursive, emitted by the new cgit-http-directives (the OS
passes it as nginx extra-content; %cgit-rate-limit-http-directives is
kept as the default-config value). The trusted-proxies default of
127.0.0.1 matches the host proxy, and set to '() when an instance is
exposed directly so a client cannot pick its own bucket.
Also for anonymous reads:
* the expensive zone goes from 5r/m burst 3 to 60r/m burst 20 -- one
request per second per client still stops history walks but leaves
log -> commit -> diff chains alone;
* /plain/ (raw file content: curl, wget, scripts) can skip the Anubis
challenge (plain-anubis-bypass?), which a tool cannot solve, and can
be restricted to user-agent prefixes (plain-user-agents) so it cannot
be used from a browser;
* internal-listen makes the Anubis-mode internal cgit address
configurable, since a --network container shares the host netns and
port 8081 may be taken.
The container uses nginx-realip (guix-bioinformatics, gn packages web):
the stock nginx is built without the real_ip module and refuses the
directives with `unknown directive "set_real_ip_from"'.
|
|
set_real_ip_from / real_ip_header / real_ip_recursive let nginx
recover the client address from X-Forwarded-For when it runs behind a
reverse proxy, so directives keyed on $binary_remote_addr stay
per-client instead of per-proxy. nginx-map emits a map block, used for
the user-agent allowlists that need a variable rather than a hand-built
string.
|
|
|
|
|
|
|
|
Job execution under slurm-api works now. The cluster-wide
AuthType=auth/jwt was the bug: auth_jwt only loads the verification key
in slurmctld/slurmdbd, so slurmd could not decode the credential and
every slurmctld -> slurmd launch RPC arrived as SLURM_AUTH_NOBODY (uid
99, "Security violation") -- which is why submissions were recorded but
never ran.
Wire it the way Slurm expects instead:
- AuthType=auth/munge for the daemon plane (slurmd verifies by uid),
- AuthAltTypes=auth/jwt so slurmctld alone also accepts JWT-signed REST
RPCs (the plugin loader reads AuthAltTypes only in slurmctld/slurmdbd),
- SLURM_JWT in slurmrestd's environment only, which makes auth_g_init()
pick auth/jwt for that process so rest_auth/jwt's per-request identity
switch (always on the primary plugin) is accepted; the caller's token
is forwarded and slurmctld decodes it to the real uid.
wait-for-munge-then-exec grew a positional JWT? flag (Guile's define*
does not take a rest arg after #:key), so only slurmrestd gets the
token. Also link /bin/bash and /bin/sh at activation: slurm runs a batch
script through its shebang and ravanan submits "#!/bin/bash" + the store
script, which a minimal Guix System cannot resolve.
Verified: ravanan's slurm-api batch system submits and completes real
slurm jobs in this container, for a single-step and a two-step workflow.
The session notes (slurm-session.md, kept out of git by a *session.md
ignore rule) document the resolution and the repro.
|
|
|
|
|
|
|
|
|
|
gn-uploader's `scripts` package moved to `uploader/scripts` (now
`uploader.scripts`), so the gunicorn app module for the gn-uploader service is
now `uploader.scripts.qcapp_wsgi:app`.
|
|
|
|
|
|
Document the GeneCup container as the worked example for the containers
built with 'guix system container' and started through the
run-guix-system-container daemon, since that is where the failures have
actually been found and where nothing was written down:
- the three -L checkouts and the two host layouts, plus the trap that a
wrong -L reports "genecup-service-type: unbound variable" and nothing
else, with the guix repl one-liner that shows the real error;
- --expose of the Gemini key and why the activation copies it (owner and
0400 mode that server.py insists on, secret never in the store);
- starting through the daemon, boot log via --log, and ssh access;
- the log map: /var/log/genecup.log (app, including the per-attempt API
errors and traceback), /var/log/gemini-check.log (boot check calling
the model), /var/log/messages, the credential file's owner/mode;
- the buffering trap (PYTHONUNBUFFERED) and that only successful
classifications are cached;
- what "Gemini classification not available" now says in each of its
three cases, and a curl one-liner that tests the model with the key;
- the container test script and its --expect-missing mode.
|
|
Shepherd redirects the app's stdout to #:log-file ("/var/log/genecup.log"),
so Python block-buffers it and the diagnostics that explain a failed
request ("Gemini batch attempt 1/3 failed: ...", "Gemini response: ...")
can trail the page by kilobytes -- the log looks empty exactly when it is
needed. Set PYTHONUNBUFFERED=1 in the service environment.
|
|
A key can list gemini-3-flash-preview in /v1beta/models and still not be
allowed to call it (preview gating, project/quota/region limits). In
that case the app's stress view prints "All related sentences (Gemini
classification not available or no specific stress types found)" after
three failed attempts, which reads like "nothing to classify" and hides
the API's own error -- exactly the report from a second deployment
where the very same key/container works elsewhere.
Make gemini-check do what the app does: POST
/v1beta/models/<model>:generateContent with a one-sentence
classification prompt (a handful of tokens), and on failure quote the
API's response body so the boot log names the reason (model not found,
permission denied, quota, TLS, ...). The model is a single definition
(%gemini-model, gemini-3-flash-preview) that must stay in sync with
server.py's hard-coded batch classifier, and the docstring says so.
Verdict is now
Gemini classification available: key in PATH works, MODEL answered (HTTP 200)
Gemini classification not available: MODEL call failed (HTTP 404): {...}
so the container tests assert "answered (HTTP 200)".
|
|
The app reads its key from ~/.config/gemini/credentials of the genecup
user (i.e. <datadir>/.config/gemini/credentials) and exits when the
mode is not 0400; when the file is missing it warns and keeps running
without LLM features. Nothing brought the host's key into the
container, and nothing said whether a key that was there actually
worked before a user hit the Gemini path in the UI.
New genecup-configuration field 'gemini-credentials': the path where
the launcher exposes the host key read-only (a secret, so never in the
store). The activation copies it to <datadir>/.config/gemini/credentials
with owner genecup and mode 0400 -- a bind mount would keep the host
owner, which the genecup user cannot read -- and complains when it is
not exposed.
New one-shot shepherd service 'gemini-check' reports at boot, in the
app's own wording, whether the key exists, has mode 0400, is non-empty
and is *accepted by the API* (GET /v1beta/models costs no tokens: 200
usable, 400/403 rejected). Verdict goes to the boot log and
/var/log/gemini-check.log. It is a program-file rather than a start
gexp because use-modules inside a shepherd start value does not take
effect (it compiles to an unbound-variable failure at runtime), and it
sets SSL_CERT_FILE since bare curl in a minimal container fails
verification with "curl: (60) ... (CAfile: none)".
genecup-container.scm exposes the host key and documents the flag; it
also documents the -L layouts and the trap found on a second host: a
wrong -L makes (forge anubis) unresolvable, so (genenetwork services
genecup) compiles to nothing and guix only reports
"genecup-service-type: unbound variable".
New test genenetwork/services/test/run_genecup_container_tests.sh
builds with the key exposed, starts through the daemon and asserts the
file owner/mode, the check verdict, GEMINI_API_KEY=set in the app log
and the app serving on 4202; --expect-missing inverts it for the
unkeyed container. Checkout locations are auto-detected, so it also
runs on flat layouts (balg01 ~/services/genecup-service). Both modes
pass here.
|
|
The modern pluto-notebook installs its Pluto/PlutoSliderServer
environment with Pkg on first run, so Pkg must be online (the
nss-certs bundle already lets it reach the registry over HTTPS). A
mounted/persistent depot makes subsequent runs offline.
|
|
The 'gn-machines' repo is defined as a channel in its own right, with the
`(directory "guix")` explicitly specifying which part of the repository to use
for the channel.
Specifying the `--load-path=./guix/` again on the CLI invocation
short-circuits that and ensures that you can only ever use the most current
local checkout -- preventing the use of different profiles (and profile
generations) for building the containers.
|
|
|
|
|
|
|
|
|
|
|
|
PlutoSliderServer on this stack can abort and be restarted by the
container's wrapper, so the server is briefly unavailable in cycles.
Make the HTTP client retry connection errors (and poll) until the
server answers, so the tests are robust to that.
|
|
Set JULIA_PKG_OFFLINE=true for the notebook server: the packaged
dependencies are already on the read-only JULIA_LOAD_PATH, and letting
Pkg hit the registry crashed the server mid-run (a libcurl
"curlmultiassign" error). Notebooks whose packages are not packaged now
fail cleanly instead of taking the server down.
Add julia_notebook_http_tests.py -- the index is served, the notebooks
are listed, and the power notebook runs and is served as power.html --
and run_julia_notebook_container_tests.sh to drive the
build/start/test/log-harvest cycle.
|
|
genenetwork/services/julia-notebook.scm: a julia-notebook service-type
running pluto-notebook's runsliderserver as an unprivileged 'julia'
user, with a configurable port, notebooks directory and Julia depot
(JULIA_DEPOT_PATH, a single dir or a colon-separated stack), and the
nss-certs CA bundle (SSL_CERT_FILE/CURL_CA_BUNDLE) so Julia's HTTP
stack can reach the package registry.
genenetwork/containers/julia-notebook-container.scm: the system
container around that service -- PlutoSliderServer on port 8890, ssh
on 2225 -- following the other containers' conventions (container-ssh,
no-op sysctl/udev, no guix service). Host mappings (the Julia depot,
real notebooks) are documented as --share/--expose at build time.
The copied specials/pluto.scm it supersedes is removed.
|
|
|
|
|
|
|
|
|