| Age | Commit message (Collapse) | Author |
|
The boot banner was displayed to stdout, which at boot is shepherd's own port
and reaches nobody: measured, the container's /var/log/messages gains
"Service container-banner has been started" but not the banner text, and
`herd start container-banner` prints it only because the client is attached to
the socket. So the text is now also written to a file the container owns,
/var/log/container-banner.log, which the daemon's LOG verb reads back through
/proc/<pid>/root (run-guix-container a18ac68). Verified on a rebuilt cgit
container: the file appears four seconds after start with the full text.
|
|
cgit showed the shape, this applies it to the rest: bnw, genecup,
gn-guile-webserver, julia-notebook, jbrowse2, slurm and gn4ai. The boot line
moved into container-ssh-service, so a container opts in with three things --
one banner string, `(issue %x-banner)` and `#:banner %x-banner` -- and the
helper does the rest: `Banner /etc/issue.net` in sshd_config (so the name is
shown BEFORE authentication), /etc/issue.net and /etc/motd for after login, and
a one-shot container-banner shepherd service that prints the same text as the
first line of the boot log. cgit's own copy of that service is gone, because
two copies would be two texts to keep right.
slurm does not use the helper (it configures sshd itself, with password
authentication for its cluster), so it gets the same four things by hand:
issue, extra-content "Banner /etc/issue.net", the two banner files, and the boot
service.
Each text names the container, what it serves and its ports, and says that it
is a container so a shell that behaves differently from the host is explained
rather than mysterious:
bnw-container BNW web app :8881, ssh :2222
genecup-container genecup web :4201, ssh :2223
gn-guile-webserver nginx :8091 -> :8092, ssh :2225
julia-notebook Pluto :8890, ssh :2225
jbrowse2-container JBrowse 2 :8080, ssh :2225
slurm-container slurmrestd :6820, ssh :2222
gn4ai-container GN4 API :3003, ssh :2223
cgit-container cgit :80, ssh :2224
Verified per file with guix repl -- issue set and sshd carrying the Banner
directive -- and with `guix system container -d --network' for all eight, then
by building cgit: its /etc/issue, /etc/motd and the compiled
shepherd-container-banner in shepherd.conf all carry the text.
One paren bug worth noting for whoever edits these next: my first pass at slurm
closed the services list one paren early, and Guix answered with
"operating-system: invalid field s" -- a record field error, nowhere near the
bracket that was wrong.
|
|
The node declared CPUs=2, which is not the container's capacity: the
container sees all 96 host CPUs (affinity 0-95) and 754 GB of RAM. Measured
with a 30-job probe, the old declaration ran 2 at a time; ravanan's LOCO
workflow (20 chromosomes x kinship+gwa) took 102 s cold against 14.3 s
single-machine.
Now: CPUs=30, RealMemory=65536, DefMemPerCPU=1024 (so a job that requests no
memory no longer takes the whole node's), SlurmdParameters=config_overrides
(the conf, not slurmd's probe, decides the node's size), and a one-second
scheduler/backfill cycle so a 40-job scatter starts in one go.
Measured after: 30 of 30 plain jobs RUNNING within a second, and ravanan's
cold LOCO run down from 102 s to 24.7 s with the same 21 051 rows.
Two things learned the hard way, both now in the comments:
* takes the *host* path. Passing the
container-side spelling (/tmp/ai-root/slurm-share) drops the mount
silently, and every job then dies with signal 53 and no stdout/stderr --
which looks like a Slurm bug, not a missing mount.
* a node that declares fewer CPUs than it has is a lie that costs 4x, but
declaring more than it has does not oversubscribe the hardware: the CPU
count is what Slurm *schedules* on, and the kernel still runs the jobs.
|
|
Comments only; the gn4ai container derivation is byte-identical before and
after (bfmdmdznzs2p0wa65z31zpcwzmq0wdk4-run-container.drv), which is the
check that nothing but prose moved.
gn4ai.scm's header now says why a batch scheduler is not in it: the
controller, the compute daemon and munge would join a public webserver's
failure domain; slurm, slurmrestd and munge accounts would enter a
container that has none; and every slurm configuration change would become
a webserver rebuild. GN4 needs the client half -- ravanan and ccwl -- and a
--slurm-api-endpoint, which is the production shape anyway: the cluster is
another machine, not a neighbour process. It also records the argument in
the other direction, since it is real: run-guix-container supervises one
container at a time, so the two cannot be up together on one host -- which
is a reason to fix the daemon or use a second box, not to merge them.
specials/slurm-container.scm says the same from its own side, and that it
deploys on a machine of its own as it stands.
|
|
waiting on backfill
Slurm's default scheduler and backfill cadence is tens of seconds: a job
that is ready to run still waits for the next cycle. For a ravanan
workflow, which submits its steps a few seconds apart, that wait
dominates the run -- six one-second jobs took about three minutes.
sched_interval=5, sched_min_interval=1, bf_interval=5 and
bf_max_job_start=40 cut it to the job's own cost, which is what makes the
20-chromosome LOCO mapping finish in ~4.5 minutes rather than tens.
For a throwaway test container, not a deployment: a real cluster's cadence
is a shared policy.
|
|
|
|
|
|
Job execution under slurm-api works now. The cluster-wide
AuthType=auth/jwt was the bug: auth_jwt only loads the verification key
in slurmctld/slurmdbd, so slurmd could not decode the credential and
every slurmctld -> slurmd launch RPC arrived as SLURM_AUTH_NOBODY (uid
99, "Security violation") -- which is why submissions were recorded but
never ran.
Wire it the way Slurm expects instead:
- AuthType=auth/munge for the daemon plane (slurmd verifies by uid),
- AuthAltTypes=auth/jwt so slurmctld alone also accepts JWT-signed REST
RPCs (the plugin loader reads AuthAltTypes only in slurmctld/slurmdbd),
- SLURM_JWT in slurmrestd's environment only, which makes auth_g_init()
pick auth/jwt for that process so rest_auth/jwt's per-request identity
switch (always on the primary plugin) is accepted; the caller's token
is forwarded and slurmctld decodes it to the real uid.
wait-for-munge-then-exec grew a positional JWT? flag (Guile's define*
does not take a rest arg after #:key), so only slurmrestd gets the
token. Also link /bin/bash and /bin/sh at activation: slurm runs a batch
script through its shebang and ravanan submits "#!/bin/bash" + the store
script, which a minimal Guix System cannot resolve.
Verified: ravanan's slurm-api batch system submits and completes real
slurm jobs in this container, for a single-step and a two-step workflow.
The session notes (slurm-session.md, kept out of git by a *session.md
ignore rule) document the resolution and the repro.
|
|
|
|
|
|
|
|
|
|
|
|
|