about summary refs log tree commit diff
diff options
context:
space:
mode:
authorPjotr Prins <pjotr.public01@thebird.nl>2026-11-16 00:41:19 +0000
committerPjotr Prins <pjotr.public01@thebird.nl>2026-11-16 00:41:19 +0000
commit39e091835268f0bc38df3efc7fe4c94dd5e7d1d7 (patch)
treef4874a8e146375805c5ccc66998e23eb1d4be8a9
parent7e52abdde8d36cb2f805d2c2d40aa123e9d168c1 (diff)
downloadgn-machines-main.tar.gz
slurm-container: 30 CPUs, honest memory, and a one-second scheduler HEAD main
The node declared CPUs=2, which is not the container's capacity: the
container sees all 96 host CPUs (affinity 0-95) and 754 GB of RAM. Measured
with a 30-job probe, the old declaration ran 2 at a time; ravanan's LOCO
workflow (20 chromosomes x kinship+gwa) took 102 s cold against 14.3 s
single-machine.

Now: CPUs=30, RealMemory=65536, DefMemPerCPU=1024 (so a job that requests no
memory no longer takes the whole node's), SlurmdParameters=config_overrides
(the conf, not slurmd's probe, decides the node's size), and a one-second
scheduler/backfill cycle so a 40-job scatter starts in one go.

Measured after: 30 of 30 plain jobs RUNNING within a second, and ravanan's
cold LOCO run down from 102 s to 24.7 s with the same 21 051 rows.

Two things learned the hard way, both now in the comments:

*  takes the *host* path. Passing the
  container-side spelling (/tmp/ai-root/slurm-share) drops the mount
  silently, and every job then dies with signal 53 and no stdout/stderr --
  which looks like a Slurm bug, not a missing mount.
* a node that declares fewer CPUs than it has is a lie that costs 4x, but
  declaring more than it has does not oversubscribe the hardware: the CPU
  count is what Slurm *schedules* on, and the kernel still runs the jobs.
-rw-r--r--specials/slurm-container.scm27
1 files changed, 23 insertions, 4 deletions
diff --git a/specials/slurm-container.scm b/specials/slurm-container.scm
index f50ce28..8765c3d 100644
--- a/specials/slurm-container.scm
+++ b/specials/slurm-container.scm
@@ -196,11 +196,24 @@ which slurmd cannot decode (it never loads the JWT key)."
                ;; (a job that is ready to run still waits for the next
                ;; cycle), which for a workflow that submits its steps a
                ;; few seconds apart becomes the dominant cost of a whole
-               ;; run: 6 one-second jobs took ~3 minutes. These settings
-               ;; are for a throwaway test container, not a deployment.
-               "SchedulerParameters=sched_interval=5,sched_min_interval=1,bf_interval=5,bf_max_job_start=40\n"
+               ;; run: 6 one-second jobs took ~3 minutes. With capacity
+               ;; for 30 jobs the cycle length is what decides whether a
+               ;; 40-job scatter starts in one go or trickles, so the
+               ;; cycle is a second and one cycle may start them all.
+               ;; These settings are for a throwaway test container, not a
+               ;; deployment.
+               "SchedulerParameters=sched_interval=1,sched_min_interval=1,bf_interval=1,bf_max_job_start=60\n"
                "SelectType=select/cons_tres\n"
                "SelectTypeParameters=CR_Core\n"
+               ;; One job = one core = 1 GiB reserved. Without a
+               ;; DefMemPerCPU a job that requests no memory is allocated
+               ;; the node's whole memory, which queues the next job
+               ;; behind it however many cores are free.
+               "DefMemPerCPU=1024\n"
+               ;; The conf, not slurmd's own probe, decides the node's
+               ;; size: the container sees the host's cores and memory,
+               ;; and slurmd refuses to start when the two disagree.
+               "SlurmdParameters=config_overrides\n"
                "ReturnToService=2\n"
                "MpiDefault=none\n"
                "\n"
@@ -219,7 +232,13 @@ which slurmd cannot decode (it never loads the JWT key)."
                ;; multi-job workflow.
                "MinJobAge=86400\n"
                "\n"
-               "NodeName=slurm-container NodeAddr=127.0.0.1 CPUs=2 RealMemory=1024 State=UNKNOWN\n"
+               ;; 30 CPUs because that is what the scatter needs: measured
+               ;; with CPUs=2, a 30-job probe ran 2 at a time and ravanan's
+               ;; 40-job LOCO workflow (20 chromosomes x kinship+gwa) took
+               ;; 102 s cold against 14.3 s single-machine. RealMemory is
+               ;; generous because the container uses the host's RAM; the
+               ;; per-job share is DefMemPerCPU above, not this number.
+               "NodeName=slurm-container NodeAddr=127.0.0.1 CPUs=30 RealMemory=65536 State=UNKNOWN\n"
                "PartitionName=debug Nodes=slurm-container Default=YES MaxTime=INFINITE State=UP\n")))
 
 ;; The JWT key is fixed, not random: `scontrol token` cannot mint one