fix(runner-manager): clean up VMs in error state immediately#802
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the runner cleanup decision logic so that platform runners backed by cloud VMs in VMState.ERROR are cleaned up immediately, rather than waiting for the creation timeout, improving recovery time and freeing capacity sooner.
Changes:
- Add an “immediate cleanup” rule for requested runners whose mapped VM is in
VMState.ERROR. - Extend unit test coverage to include an ERROR-state VM cleanup scenario.
- Bump
github-runner-managerversion and document the user-visible behavior change in the changelog.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| github-runner-manager/src/github_runner_manager/manager/runner_manager.py | Adds ERROR-state VM cleanup rule and updates cleanup selection logic. |
| github-runner-manager/tests/unit/manager/test_runner_manager.py | Adds a unit test case asserting immediate cleanup for ERROR-state VMs. |
| github-runner-manager/tests/unit/factories/runner_instance_factory.py | Extends cloud runner factory helper to allow specifying VM state for tests. |
| github-runner-manager/pyproject.toml | Bumps application version to 0.18.2. |
| docs/changelog.md | Records the new immediate cleanup behavior for ERROR-state VMs. |
Cleanup decisions only considered platform health and VM age, never the cloud VM state. A VM reported as ERROR by OpenStack never recovers, yet it kept occupying a runner slot for the full creation timeout (~23 minutes) before being collected, and no replacement was created in the meantime. Treat runners whose cloud VM is in ERROR state as eligible for cleanup regardless of age, freeing the slot for a replacement on the next reconcile.
d6133aa to
b32cb18
Compare
yanksyoon
approved these changes
Jun 26, 2026
weiiwang01
approved these changes
Jun 26, 2026
javierdelapuente
approved these changes
Jun 26, 2026
… python 3.12 The unit job ran on a self-hosted runner whose image now provides Python 3.14. With the open-ended requires-python (>=3.10), setup-python/uv resolves to the newest interpreter, and Python 3.14 changed the default multiprocessing start method on Linux from fork to forkserver. forkserver pickles the Process target, which test_flush_runner_locked cannot do (FlaskClient.post references an unpicklable Flask init lambda), so the unit job failed. Switch the job to GitHub-hosted ubuntu-24.04 (Noble) and pin Python to 3.12 so tox uses a fork-default interpreter. The integration job stays on self-hosted runners as it requires OpenStack.
Collaborator
Author
|
Force-merged (admin) despite the failing The failure is a pre-existing flaky test unrelated to this PR:
Follow-up to de-flake the test: #803. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Cleans up runners whose cloud VM is in an
ERRORstate immediately, instead of waiting for the creation timeout._get_platform_runners_to_cleanuppreviously decided cleanup purely from platform health (online/busy/deletable) plus VM age, never consulting the cloudVMState. An ERROR VM was therefore treated like one still booting and only collected once older thanRUNNER_MAXIMUM_CREATION_TIME(~23 min). This adds an explicit rule: any requested runner whose mapped VM hasstate == VMState.ERRORis eligible for cleanup regardless of age.Why we need it
An OpenStack VM reported as
ERRORnever recovers, but it still counted toward the desired runner total, so the slot stayed occupied with a dead VM for ~23 minutes and no replacement was created until cleanup finally evicted it. This leaves a flavor with zero usable capacity for that whole window.Checklist
docs/changelog.mddocs/changelog.mdwith user-relevant changesterraform fmtpasses andtflintreports no errors — n/agithub-runner-manager/pyproject.toml. — bumped 0.18.1 → 0.18.2