diff options
| author | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-11-16 00:41:19 +0000 |
|---|---|---|
| committer | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-11-16 00:41:19 +0000 |
| commit | 39e091835268f0bc38df3efc7fe4c94dd5e7d1d7 (patch) | |
| tree | f4874a8e146375805c5ccc66998e23eb1d4be8a9 /specials | |
| parent | 7e52abdde8d36cb2f805d2c2d40aa123e9d168c1 (diff) | |
| download | gn-machines-main.tar.gz | |
The node declared CPUs=2, which is not the container's capacity: the container sees all 96 host CPUs (affinity 0-95) and 754 GB of RAM. Measured with a 30-job probe, the old declaration ran 2 at a time; ravanan's LOCO workflow (20 chromosomes x kinship+gwa) took 102 s cold against 14.3 s single-machine. Now: CPUs=30, RealMemory=65536, DefMemPerCPU=1024 (so a job that requests no memory no longer takes the whole node's), SlurmdParameters=config_overrides (the conf, not slurmd's probe, decides the node's size), and a one-second scheduler/backfill cycle so a 40-job scatter starts in one go. Measured after: 30 of 30 plain jobs RUNNING within a second, and ravanan's cold LOCO run down from 102 s to 24.7 s with the same 21 051 rows. Two things learned the hard way, both now in the comments: * takes the *host* path. Passing the container-side spelling (/tmp/ai-root/slurm-share) drops the mount silently, and every job then dies with signal 53 and no stdout/stderr -- which looks like a Slurm bug, not a missing mount. * a node that declares fewer CPUs than it has is a lie that costs 4x, but declaring more than it has does not oversubscribe the hardware: the CPU count is what Slurm *schedules* on, and the kernel still runs the jobs.
Diffstat (limited to 'specials')
| -rw-r--r-- | specials/slurm-container.scm | 27 |
1 files changed, 23 insertions, 4 deletions
diff --git a/specials/slurm-container.scm b/specials/slurm-container.scm index f50ce28..8765c3d 100644 --- a/specials/slurm-container.scm +++ b/specials/slurm-container.scm @@ -196,11 +196,24 @@ which slurmd cannot decode (it never loads the JWT key)." ;; (a job that is ready to run still waits for the next ;; cycle), which for a workflow that submits its steps a ;; few seconds apart becomes the dominant cost of a whole - ;; run: 6 one-second jobs took ~3 minutes. These settings - ;; are for a throwaway test container, not a deployment. - "SchedulerParameters=sched_interval=5,sched_min_interval=1,bf_interval=5,bf_max_job_start=40\n" + ;; run: 6 one-second jobs took ~3 minutes. With capacity + ;; for 30 jobs the cycle length is what decides whether a + ;; 40-job scatter starts in one go or trickles, so the + ;; cycle is a second and one cycle may start them all. + ;; These settings are for a throwaway test container, not a + ;; deployment. + "SchedulerParameters=sched_interval=1,sched_min_interval=1,bf_interval=1,bf_max_job_start=60\n" "SelectType=select/cons_tres\n" "SelectTypeParameters=CR_Core\n" + ;; One job = one core = 1 GiB reserved. Without a + ;; DefMemPerCPU a job that requests no memory is allocated + ;; the node's whole memory, which queues the next job + ;; behind it however many cores are free. + "DefMemPerCPU=1024\n" + ;; The conf, not slurmd's own probe, decides the node's + ;; size: the container sees the host's cores and memory, + ;; and slurmd refuses to start when the two disagree. + "SlurmdParameters=config_overrides\n" "ReturnToService=2\n" "MpiDefault=none\n" "\n" @@ -219,7 +232,13 @@ which slurmd cannot decode (it never loads the JWT key)." ;; multi-job workflow. "MinJobAge=86400\n" "\n" - "NodeName=slurm-container NodeAddr=127.0.0.1 CPUs=2 RealMemory=1024 State=UNKNOWN\n" + ;; 30 CPUs because that is what the scatter needs: measured + ;; with CPUs=2, a 30-job probe ran 2 at a time and ravanan's + ;; 40-job LOCO workflow (20 chromosomes x kinship+gwa) took + ;; 102 s cold against 14.3 s single-machine. RealMemory is + ;; generous because the container uses the host's RAM; the + ;; per-job share is DefMemPerCPU above, not this number. + "NodeName=slurm-container NodeAddr=127.0.0.1 CPUs=30 RealMemory=65536 State=UNKNOWN\n" "PartitionName=debug Nodes=slurm-container Default=YES MaxTime=INFINITE State=UP\n"))) ;; The JWT key is fixed, not random: `scontrol token` cannot mint one |
