Skip to content

[bugfix] Fix Slurm job array completion detection - #3707

Open
dingp wants to merge 1 commit into
reframe-hpc:developfrom
dingp:dingpf/fix-slurm-array-completion-detection
Open

[bugfix] Fix Slurm job array completion detection#3707
dingp wants to merge 1 commit into
reframe-hpc:developfrom
dingp:dingpf/fix-slurm-array-completion-detection

Conversation

@dingp

@dingp dingp commented Aug 8, 2026

Copy link
Copy Markdown

Summary

  • recognize compact Slurm array IDs containing lists, ranges, steps and percent throttles
  • track the requested array task count from the --array option
  • keep an array nonterminal when a scheduler snapshot accounts for fewer tasks than requested
  • add scheduler regression tests for incomplete and compact sacct responses

Background

Slurm can report a pending throttled array element as 56502501_[1%1]|PENDING. The previous job-ID pattern did not match that form. If sacct simultaneously reported an earlier element as completed, ReFrame could treat the array as complete and merge output before the remaining element ran.

Testing

  • venv/bin/python test_reframe.py -q unittests/test_schedulers.py -k "slurm_count_array_tasks or slurm_array_jobid_pattern or slurm_poll_job_array" — 12 passed, 27 skipped
  • venv/bin/python test_reframe.py -q unittests/test_schedulers.py — 166 passed, 226 skipped

Signed-off-by: Pengfei Ding <9438483+dingp@users.noreply.github.com>
@dingp
dingp marked this pull request as ready for review August 8, 2026 18:55
Copilot AI lite review requested due to automatic review settings August 8, 2026 18:55

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes ReFrame’s Slurm job-array completion detection when Slurm reports compact array element IDs (including lists/ranges/steps and percent throttles). It ensures a job array is not treated as complete (and its outputs merged) until the scheduler snapshot accounts for all tasks requested via --array.

Changes:

  • Extend Slurm job-id matching to recognize compact array element expressions (e.g., 123_[1,3-7:2%2]).
  • Track the requested array task count from the --array option and keep arrays nonterminal when fewer tasks are observed in scheduler snapshots.
  • Add unit tests covering array task counting, job-id pattern matching, and polling behavior with incomplete/compact sacct outputs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
reframe/core/schedulers/slurm.py Adds array-expression task counting, broadens Slurm array job-id pattern matching, and prevents premature completion when fewer tasks are observed than requested.
unittests/test_schedulers.py Adds regression tests for array task counting, compact job-id patterns, and polling behavior with incomplete/compact sacct responses.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants