about summary refs log tree commit diff
path: root/specials
AgeCommit message (Collapse)Author
3 dayscontainers: write the banner to /var/log/container-banner.log as wellPjotr
The boot banner was displayed to stdout, which at boot is shepherd's own port and reaches nobody: measured, the container's /var/log/messages gains "Service container-banner has been started" but not the banner text, and `herd start container-banner` prints it only because the client is attached to the socket. So the text is now also written to a file the container owns, /var/log/container-banner.log, which the daemon's LOG verb reads back through /proc/<pid>/root (run-guix-container a18ac68). Verified on a rebuilt cgit container: the file appears four seconds after start with the full text.
4 dayscontainers: every container now says which one it isPjotr
cgit showed the shape, this applies it to the rest: bnw, genecup, gn-guile-webserver, julia-notebook, jbrowse2, slurm and gn4ai. The boot line moved into container-ssh-service, so a container opts in with three things -- one banner string, `(issue %x-banner)` and `#:banner %x-banner` -- and the helper does the rest: `Banner /etc/issue.net` in sshd_config (so the name is shown BEFORE authentication), /etc/issue.net and /etc/motd for after login, and a one-shot container-banner shepherd service that prints the same text as the first line of the boot log. cgit's own copy of that service is gone, because two copies would be two texts to keep right. slurm does not use the helper (it configures sshd itself, with password authentication for its cluster), so it gets the same four things by hand: issue, extra-content "Banner /etc/issue.net", the two banner files, and the boot service. Each text names the container, what it serves and its ports, and says that it is a container so a shell that behaves differently from the host is explained rather than mysterious: bnw-container BNW web app :8881, ssh :2222 genecup-container genecup web :4201, ssh :2223 gn-guile-webserver nginx :8091 -> :8092, ssh :2225 julia-notebook Pluto :8890, ssh :2225 jbrowse2-container JBrowse 2 :8080, ssh :2225 slurm-container slurmrestd :6820, ssh :2222 gn4ai-container GN4 API :3003, ssh :2223 cgit-container cgit :80, ssh :2224 Verified per file with guix repl -- issue set and sshd carrying the Banner directive -- and with `guix system container -d --network' for all eight, then by building cgit: its /etc/issue, /etc/motd and the compiled shepherd-container-banner in shepherd.conf all carry the text. One paren bug worth noting for whoever edits these next: my first pass at slurm closed the services list one paren early, and Guix answered with "operating-system: invalid field s" -- a record field error, nowhere near the bracket that was wrong.
4 daysslurm-container: 30 CPUs, honest memory, and a one-second schedulerPjotr Prins
The node declared CPUs=2, which is not the container's capacity: the container sees all 96 host CPUs (affinity 0-95) and 754 GB of RAM. Measured with a 30-job probe, the old declaration ran 2 at a time; ravanan's LOCO workflow (20 chromosomes x kinship+gwa) took 102 s cold against 14.3 s single-machine. Now: CPUs=30, RealMemory=65536, DefMemPerCPU=1024 (so a job that requests no memory no longer takes the whole node's), SlurmdParameters=config_overrides (the conf, not slurmd's probe, decides the node's size), and a one-second scheduler/backfill cycle so a 40-job scatter starts in one go. Measured after: 30 of 30 plain jobs RUNNING within a second, and ravanan's cold LOCO run down from 102 s to 24.7 s with the same 21 051 rows. Two things learned the hard way, both now in the comments: * takes the *host* path. Passing the container-side spelling (/tmp/ai-root/slurm-share) drops the mount silently, and every job then dies with signal 53 and no stdout/stderr -- which looks like a Slurm bug, not a missing mount. * a node that declares fewer CPUs than it has is a lie that costs 4x, but declaring more than it has does not oversubscribe the hardware: the CPU count is what Slurm *schedules* on, and the kernel still runs the jobs.
5 daysNote the topology: slurm stays a separate box from the GN4AI webserverPjotr
Comments only; the gn4ai container derivation is byte-identical before and after (bfmdmdznzs2p0wa65z31zpcwzmq0wdk4-run-container.drv), which is the check that nothing but prose moved. gn4ai.scm's header now says why a batch scheduler is not in it: the controller, the compute daemon and munge would join a public webserver's failure domain; slurm, slurmrestd and munge accounts would enter a container that has none; and every slurm configuration change would become a webserver rebuild. GN4 needs the client half -- ravanan and ccwl -- and a --slurm-api-endpoint, which is the production shape anyway: the cluster is another machine, not a neighbour process. It also records the argument in the other direction, since it is real: run-guix-container supervises one container at a time, so the two cannot be up together on one host -- which is a reason to fix the daemon or use a second box, not to merge them. specials/slurm-container.scm says the same from its own side, and that it deploys on a machine of its own as it stands.
5 daysslurm-container: tight scheduler cycles, so a scattered workflow is not ↵Pjotr Prins
waiting on backfill Slurm's default scheduler and backfill cadence is tens of seconds: a job that is ready to run still waits for the next cycle. For a ravanan workflow, which submits its steps a few seconds apart, that wait dominates the run -- six one-second jobs took about three minutes. sched_interval=5, sched_min_interval=1, bf_interval=5 and bf_max_job_start=40 cut it to the job's own cost, which is what makes the 20-chromosome LOCO mapping finish in ~4.5 minutes rather than tens. For a throwaway test container, not a deployment: a real cluster's cadence is a shared policy.
2026-10-28SlurmPjotr Prins
2026-10-28JBrowse2 containerPjotr Prins
2026-10-28slurm-container: AuthAltTypes for the REST plane, munge for the daemons.Pjotr
Job execution under slurm-api works now. The cluster-wide AuthType=auth/jwt was the bug: auth_jwt only loads the verification key in slurmctld/slurmdbd, so slurmd could not decode the credential and every slurmctld -> slurmd launch RPC arrived as SLURM_AUTH_NOBODY (uid 99, "Security violation") -- which is why submissions were recorded but never ran. Wire it the way Slurm expects instead: - AuthType=auth/munge for the daemon plane (slurmd verifies by uid), - AuthAltTypes=auth/jwt so slurmctld alone also accepts JWT-signed REST RPCs (the plugin loader reads AuthAltTypes only in slurmctld/slurmdbd), - SLURM_JWT in slurmrestd's environment only, which makes auth_g_init() pick auth/jwt for that process so rest_auth/jwt's per-request identity switch (always on the primary plugin) is accepted; the caller's token is forwarded and slurmctld decodes it to the real uid. wait-for-munge-then-exec grew a positional JWT? flag (Guile's define* does not take a rest arg after #:key), so only slurmrestd gets the token. Also link /bin/bash and /bin/sh at activation: slurm runs a batch script through its shebang and ravanan submits "#!/bin/bash" + the store script, which a minimal Guix System cannot resolve. Verified: ravanan's slurm-api batch system submits and completes real slurm jobs in this container, for a single-step and a two-step workflow. The session notes (slurm-session.md, kept out of git by a *session.md ignore rule) document the resolution and the repro.
2026-10-22slurm container (WIP)Pjotr Prins
2026-10-02Move unused filesPjotr Prins
2025-10-04Prevent gn-auth-port from conflictingPjotr Prins
2024-04-05Document configuration settingsPjotr Prins
2024-04-04Remove tmp before creating itPjotr Prins
2024-04-03gendev: special machine for PjotrPjotr Prins