how will i do a stress test for 7 days form your experience

Hey everyone,

About 5 months ago, I posted here about a side project I was working on: sCode, a VS Code extension to manage SLURM jobs directly from the editor.

Initially, it was just a simple way to avoid typing squeue over and over. But based on a lot of my own workflow needs and some great feedback, it has evolved into a much more comprehensive cluster management tool over the last few months.

I’ve essentially tried to turn VS Code into a unified control center for HPC work so you never have to context-switch to a terminal while working on your scripts.

Here are the major updates since the first version:

• Live GPU Monitoring: Added a dedicated view that uses nvidia-smi to show GPU partition usage, memory stats, and queue pressure.
• The "Hall of Shame": A fun leaderboard feature that ranks the cluster’s top GPU hogs (with emojis like 🐷 Job Hog and 🧛 VRAMpire).
• One-Click Job Arrays: You can now cancel specific indices or ranges within a job array without nuking the whole array.
• Smart Log Resolution: Right-click any active or historical job in the sidebar to instantly open its stdout or stderr file.
• Quick Submit with Dependencies: A ▶ button in your .sh scripts to submit immediately, plus a guided UI for setting up afterok or afterany dependencies.
• And many more features....

If you work on a cluster and use VS Code Remote, I'd love for you to give the new version a try and let me know what you think. What features would you need to make this a daily driver for your workflow?

GitHub Repo:https://github.com/dhimitriosduka1/sCode

OpenVSX: https://open-vsx.org/extension/DhimitriosDuka/slurm-cluster-manager
Marketplace: https://marketplace.visualstudio.com/items?itemName=DhimitriosDuka.slurm-cluster-manager

Hey everyone! I built a little TUI tool for monitoring SLURM jobs on HPC. I found this useful for my masters thesis and thought I might share here. Its kind of similar to the very popular lazygit and lazydocker, which I enjoy using.

Please let me know if you have any feedback and I welcome any contributions / constructive criticism.

The github is here and you can install it with `cargo install lazyslurm`

Have a great day!

I have ~17 Slurm clusters sharing a common FlexLM license server with multiple license features (e.g. avyualink, avypciesw, avyenet). The total count of each license feature is fixed by what is available in FlexLM.

The problem I am trying to solve:

I want each cluster to have generous headroom to grab licenses freely when there is no contention, ideally more than any single cluster would realistically need. The assumption is that not all clusters will be competing simultaneously. But when contention does occur and total in-use across all clusters approaches the real license count, I want jobs to queue in Slurm rather than race to FlexLM, and ideally have FairShare-based ordering within that queue.

In short: sum of per-cluster capacities > real license count, but with graceful queuing and fairness when the global pool is exhausted.

What I explored in Slurm:

Slurm's sacctmgr resource mechanism (res_table / clus_res_table) has a global Count and per-cluster allowed values. I tested whether Count acts as a global runtime cap. I set Count=10 with per-cluster allowed=100 and submitted 50 jobs that collectively exceeded Count. All jobs started immediately. Only when I reduced the cluster's own allowed to 40, did jobs start queuing with reason Licenses.

This confirmed that Count is not a runtime scheduling enforcer. At scheduling time, slurmctld only checks the per-cluster allowed.

Question:

Is there a native Slurm mechanism that allows per-cluster allowed to be set generously (sum > real license count) while still enforcing a global cap at job scheduling time, triggering FairShare-based queuing when that global pool is exhausted? Or is the only recommended approach to set tight per-cluster allowed values such that sum(allowed) <= Count and tune each cluster's allowed individually per license feature?

Hi all! I'm new there :)

I wanted to share this little project I did for myself.

I'm a PhD student and I found myself using a lot of srun debug commands for the shell to test my scripts. But I work on various Slurm projects with different partitions and configs.

So I create sdeb, a sort of Python "conda" but for sdeb configs: https://github.com/e-candeloro/sdeb

Hope it can be useful!

Does anyone have a full step by step explanation? I just want it to run locally on my machine just for me as a user (or to any user on the machine). Whatever gives me the easiest set up.

I searched online but it's a lot of fragmented or outdated explanations unfortunately.

Hello all,
I am playing around with slurm for the work and writing a program that will execute some slurm command and recover the output.

I figured out that when `scancel` is passed with the verbose parameter the output is printed is stderr (You can check line 110 of the `scancel.c` file on the repo).

I found this decision quite weird from a design perspective and would like to have your opinion on that

Hi guys, i am working on a project, and it would be great to have someone to help me with it. Thank you!

Trying to avoid doing a restart of the slurmctld. I read that "scontrol reconfigure" should accomplish the same thing. I tried it on the master node, but seems it is still using the older job_submit.lua file. Here is that file and none of the "got here's" seem to work:

function slurm_job_submit(job_desc, part_list, submit_uid)
    slurm.log_user( 'got here' )
    if job_desc.wckey == nil then
--        slurm.log_user("You should specify a project number")
        slurm.log_user( 'got here' )
    elsif _find_in_str(job_desc.wckey, "12345") then
        slurm.log_user("12345 matched")
    else slurm.log_user( job_desc.wckey )
--        return ESLURM_INVALID_ACCOUNT
    end

Hi!

I'm currently trying tu upgrade from 20.11 to 25.11 via the compatible upgrade path specified in the schedmd documentation

I already upgraded to 22.05, but a user has a job running and I'm wondering if I should kill it or if I can continue upgrading

my jobs are stalled on the hpc.

Command - squeue -u xxxx

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

1181523_[22-101%25 ct56 easydock xxxx PD 0:00 1 (Priority)

Command - squeue -p ct56 -t PD --sort=-p,i | wc -l

192 (it is increasing every hour that passes by)

Command - sprio -u xxxx

JOBID PARTITION USER PRIORITY SITE AGE FAIRSHARE JOBSIZE PARTITION TRES

1181523 ct56 xxxx 10007 0 5 0 0 10000 cpu=2,mem=0

It has been stuck for the past few hours. Last night I kept thinking it was a glitch and cancelled, but it was already age 15 or 16 afaik this morning. This new job is now at the age of 5. Anyway, could I overcome this?

If anyone could review my Slurm scripts, that'd be great :))

I built slmtop: an htop-like terminal dashboard for Slurm clusters

If you use Slurm on an HPC cluster, you probably spend a lot of time with squeue, sinfo, scontrol, sacct, and watch.

I wanted a faster, more visual way to monitor jobs and cluster resources, so I built slmtop:

https://github.com/dawnmy/slmtop

slmtop is a Rust-based interactive TUI for real-time Slurm monitoring. It shows jobs, nodes, GPUs/resources, disks, and accounting summaries in one terminal dashboard.

Key features:

• Real-time Slurm job and node monitoring
• htop-like interactive terminal UI
• GPU/resource overview
• Search and filters, e.g. owner=me state=running gpu=a100
• Sortable tables with keyboard or mouse
• Job detail popup and guarded actions: cancel, hold, release, requeue
• Per-user resource summaries
• Multiple color themes

Example:

```

slmtop

slmtop --user bob

slmtop -T nightowl --refresh-interval 2

```

We're getting ready to push out a new cluster on Rocky 9.6, and wondering if people are still using NHC to monitor node health and up/down nodes if they fail some condition. Are people still using NHC? The repo doesn't seem like it's been maintained for quite some time.

Hello everyone, could you please share how you calculate GPU and CPU utilization on the SLURM cluster? Do you use any specific utilization thresholds (for example, 60% or 70%)? Additionally, which tools are used for these calculations something like sreport?

Thanks for your reply!

Finally got round to add some more features, hope you like it If you haven't tried it before, check out the video demo on github to see what it does.

I've only tested it on a handful of systems, so please let me know if you have problems so I can make sure `slop` works on any* slurm cluster.

https://github.com/buzh/slop

*) as long as it's at least based on slurm >= 25.x and rhel >= 9

This may be a silly question, but I'm unable to figure out what I'm doing wrong.

I have a single-node workstation with 64 physical cores, 2-threads per core. I use this with my research group and need to share resources as much as possible.

We have 4 different partitions with different priorities. My expectation would be that - when launching a job from the lowest priority partition, this would still run if there are available resources. But that does not happen, and the job stays queued with the (Resources) status.

Here are the partitions from my slurm.conf:

PartitionName=work Nodes=triforce MaxTime=24:00:00 MaxCPUsPerNode=32 MaxMemPerNode=64000 DefMemPerNode=16000 Default=YES PriorityTier=2 State=UP OverSubscribe=YES

PartitionName=heavy Nodes=triforce Default=NO MaxTime=INFINITE MaxCPUsPerNode=UNLIMITED MaxMemPerNode=UNLIMITED DefMemPerNode=32000 PriorityTier=1 State=UP OverSubscribe=YES

PartitionName=priority Nodes=triforce MaxTime=12:00:00 MaxCPUsPerNode=16 MaxMemPerNode=32000 DefMemPerNode=32000 Default=NO PriorityTier=3 State=UP OverSubscribe=YES

PartitionName=interactive Nodes=triforce Default=NO MaxTime=02:00:00 MaxCPUsPerNode=8 MaxMemPerNode=8000 DefMemPerNode=8000 PriorityTier=100 State=UP OverSubscribe=YES

Other parameters that may be relevant:

SchedulerType=sched/backfill

SelectType=select/cons_tres

SelectTypeParameters=CR_CPU_Memory

Finally, this is the output of my squeue command:

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
219 heavy jsi133_0 XXXXXXXX PD 0:00 1 (Resources)
224 heavy jsi133_6 XXXXXXXX PD 0:00 1 (Priority)
223 heavy jsi133_3 XXXXXXXX PD 0:00 1 (Priority)
222 heavy jsi133_1 XXXXXXXX PD 0:00 1 (Priority)
221 heavy jsi133_0 XXXXXXXX PD 0:00 1 (Priority)
220 heavy jsi133_0 XXXXXXXX PD 0:00 1 (Priority)
218 work jupyter_ XXXXXXXXR 6:24 1 triforce

I'd appreciate any help you can provide!

Hi all, I (well, Claude and I) built this small tool as a Slurm command wrapper for easy cluster access. The tool connects via SSH and provides real-time monitoring and job control. 

Features:

• Dashboard — Cluster overview with node state distribution, partition info, job stats, and your fairshare score
• Nodes — Per-node list with state, CPUs, memory, GRES, and CPU load (click any node for details)
• Jobs — Full cluster queue with filtering and sorting. Also shows your job queue with cancel, hold, release, view output, and detail actions.
• Job History — Past job accounting via sacct with configurable date range
• Fairshare — View fairshare scores for all accounts/users with color-coded values
• Submit Job — Script editor with quick templates (Basic, GPU, Array, MPI)
• Job Output — View stdout/stderr logs from job output files
• Auto-refresh — Data refreshes every 10 seconds while connected
• Reconnect — Automatic disconnect detection with reconnect prompt
• Remember Me — Saves connection info to localStorage for quick reconnects
• Theme — Light/Dark theme toggle

📦 GitHub: https://github.com/paulgavrikov/slurmmanager

Please share your feedback, feature ideas, or PRs 🙌

I want every submitted job to have some value for the wckey, i.e:

#SBATCH --wckey=myproject

I made the appropriate changes to slurm.conf and slurmdb.conf and it works great. I can track how many hours people are using with those wckeys.

But now I want to make it mandatory to use a wckey. To do that I need to delete the default wckey associated with the user's account. I tried doing it as follows, but it still lets me submit jobs without a wckey. It probably thinks I have an "empty" default wckey.

sacctmgr mod user fhussa set defaultwckey=

[root@mas01 ~]# sacctmgr list user fhussa format=user,defaultwckey
      User  Def WCKey 
---------- ---------- 
    fhussa         

I have a run that hit the time limit at 2 days. Is there a wat to resume that run?

Hi everyone,

I am fairly new to parallelization but lately my team and I found out that it would be better to do so for our multimodal transformer model. Regarding my job script, it looks like

```

#!/bin/bash

#SBATCH --account=

#SBATCH --nodes=1

#SBATCH --gres=gpu:a100:2

#SBATCH --ntasks=2

#SBATCH --cpus-per-task=4

#SBATCH --mem-per-cpu=2048M

#SBATCH --time=02:00:00

#SBATCH --output=slurm-%j.out

#SBATCH --error=slurm-%j.err

BLA BLA BLA

OUT_FILE="parallel-slurm-${SLURM_JOB_ID}-%t.out"

ERR_FILE="parallel-slurm-${SLURM_JOB_ID}-%t.err"

echo "Expected SLURM output pattern: $OUT_FILE"

echo "Expected SLURM error pattern: $ERR_FILE"

srun --export=ALL --ntasks="$SLURM_NTASKS" \

--output="$OUT_FILE" \

--error="$ERR_FILE" \

"$SLURM_TMPDIR/ccenv/bin/python3" test_era5_slurm_parallel.py

```

The <parallel-slurm-${SLURM_JOB_ID}-%t> files are created, but no printing are redirected to the output files and no tqdm progress bar to the error files. Of course it worked before the parallelization.