slurm-container: 30 CPUs, honest memory, and a one-second scheduler
HEAD mainThe node declared CPUs=2, which is not the container's capacity: the
container sees all 96 host CPUs (affinity 0-95) and 754 GB of RAM. Measured
with a 30-job probe, the old declaration ran 2 at a time; ravanan's LOCO
workflow (20 chromosomes x kinship+gwa) took 102 s cold against 14.3 s
single-machine.
Now: CPUs=30, RealMemory=65536, DefMemPerCPU=1024 (so a job that requests no
memory no longer takes the whole node's), SlurmdParameters=config_overrides
(the conf, not slurmd's probe, decides the node's size), and a one-second
scheduler/backfill cycle so a 40-job scatter starts in one go.
Measured after: 30 of 30 plain jobs RUNNING within a second, and ravanan's
cold LOCO run down from 102 s to 24.7 s with the same 21 051 rows.
Two things learned the hard way, both now in the comments:
* takes the *host* path. Passing the
container-side spelling (/tmp/ai-root/slurm-share) drops the mount
silently, and every job then dies with signal 53 and no stdout/stderr --
which looks like a Slurm bug, not a missing mount.
* a node that declares fewer CPUs than it has is a lie that costs 4x, but
declaring more than it has does not oversubscribe the hardware: the CPU
count is what Slurm *schedules* on, and the kernel still runs the jobs.
1 files changed, 23 insertions, 4 deletions
diff --git a/specials/slurm-container.scm b/specials/slurm-container.scm
index f50ce28..8765c3d 100644
--- a/specials/slurm-container.scm
+++ b/specials/slurm-container.scm
@@ -196,11 +196,24 @@ which slurmd cannot decode (it never loads the JWT key)."
;; (a job that is ready to run still waits for the next
;; cycle), which for a workflow that submits its steps a
;; few seconds apart becomes the dominant cost of a whole
- ;; run: 6 one-second jobs took ~3 minutes. These settings
- ;; are for a throwaway test container, not a deployment.
- "SchedulerParameters=sched_interval=5,sched_min_interval=1,bf_interval=5,bf_max_job_start=40\n"
+ ;; run: 6 one-second jobs took ~3 minutes. With capacity
+ ;; for 30 jobs the cycle length is what decides whether a
+ ;; 40-job scatter starts in one go or trickles, so the
+ ;; cycle is a second and one cycle may start them all.
+ ;; These settings are for a throwaway test container, not a
+ ;; deployment.
+ "SchedulerParameters=sched_interval=1,sched_min_interval=1,bf_interval=1,bf_max_job_start=60\n"
"SelectType=select/cons_tres\n"
"SelectTypeParameters=CR_Core\n"
+ ;; One job = one core = 1 GiB reserved. Without a
+ ;; DefMemPerCPU a job that requests no memory is allocated
+ ;; the node's whole memory, which queues the next job
+ ;; behind it however many cores are free.
+ "DefMemPerCPU=1024\n"
+ ;; The conf, not slurmd's own probe, decides the node's
+ ;; size: the container sees the host's cores and memory,
+ ;; and slurmd refuses to start when the two disagree.
+ "SlurmdParameters=config_overrides\n"
"ReturnToService=2\n"
"MpiDefault=none\n"
"\n"
@@ -219,7 +232,13 @@ which slurmd cannot decode (it never loads the JWT key)."
;; multi-job workflow.
"MinJobAge=86400\n"
"\n"
- "NodeName=slurm-container NodeAddr=127.0.0.1 CPUs=2 RealMemory=1024 State=UNKNOWN\n"
+ ;; 30 CPUs because that is what the scatter needs: measured
+ ;; with CPUs=2, a 30-job probe ran 2 at a time and ravanan's
+ ;; 40-job LOCO workflow (20 chromosomes x kinship+gwa) took
+ ;; 102 s cold against 14.3 s single-machine. RealMemory is
+ ;; generous because the container uses the host's RAM; the
+ ;; per-job share is DefMemPerCPU above, not this number.
+ "NodeName=slurm-container NodeAddr=127.0.0.1 CPUs=30 RealMemory=65536 State=UNKNOWN\n"
"PartitionName=debug Nodes=slurm-container Default=YES MaxTime=INFINITE State=UP\n")))
;; The JWT key is fixed, not random: `scontrol token` cannot mint one
|