Skip to content

fix(runner-manager): clean up VMs in error state immediately#802

Merged
cbartz merged 2 commits into
mainfrom
fix/cleanup-errored-vms
Jun 29, 2026
Merged

fix(runner-manager): clean up VMs in error state immediately#802
cbartz merged 2 commits into
mainfrom
fix/cleanup-errored-vms

Conversation

@cbartz

@cbartz cbartz commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

What this PR does

Cleans up runners whose cloud VM is in an ERROR state immediately, instead of waiting for the creation timeout.

_get_platform_runners_to_cleanup previously decided cleanup purely from platform health (online/busy/deletable) plus VM age, never consulting the cloud VMState. An ERROR VM was therefore treated like one still booting and only collected once older than RUNNER_MAXIMUM_CREATION_TIME (~23 min). This adds an explicit rule: any requested runner whose mapped VM has state == VMState.ERROR is eligible for cleanup regardless of age.

Why we need it

An OpenStack VM reported as ERROR never recovers, but it still counted toward the desired runner total, so the slot stayed occupied with a dead VM for ~23 minutes and no replacement was created until cleanup finally evicted it. This leaves a flavor with zero usable capacity for that whole window.

Checklist

  • I followed the contributing guide
  • I added or updated the documentation (if applicable) — updated docs/changelog.md
  • I updated docs/changelog.md with user-relevant changes
  • I used AI to assist with preparing this PR
  • I added or updated tests as needed (unit and integration) — added a unit case covering immediate cleanup of a recent ERROR-state VM
  • If this is a Grafana dashboard: I added a screenshot of the dashboard — n/a
  • If this is Terraform: terraform fmt passes and tflint reports no errors — n/a
  • If the github-runner-manager application has been changed: The application version number is updated in github-runner-manager/pyproject.toml. — bumped 0.18.1 → 0.18.2

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the runner cleanup decision logic so that platform runners backed by cloud VMs in VMState.ERROR are cleaned up immediately, rather than waiting for the creation timeout, improving recovery time and freeing capacity sooner.

Changes:

  • Add an “immediate cleanup” rule for requested runners whose mapped VM is in VMState.ERROR.
  • Extend unit test coverage to include an ERROR-state VM cleanup scenario.
  • Bump github-runner-manager version and document the user-visible behavior change in the changelog.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
github-runner-manager/src/github_runner_manager/manager/runner_manager.py Adds ERROR-state VM cleanup rule and updates cleanup selection logic.
github-runner-manager/tests/unit/manager/test_runner_manager.py Adds a unit test case asserting immediate cleanup for ERROR-state VMs.
github-runner-manager/tests/unit/factories/runner_instance_factory.py Extends cloud runner factory helper to allow specifying VM state for tests.
github-runner-manager/pyproject.toml Bumps application version to 0.18.2.
docs/changelog.md Records the new immediate cleanup behavior for ERROR-state VMs.

Comment thread github-runner-manager/src/github_runner_manager/manager/runner_manager.py Outdated
Comment thread github-runner-manager/src/github_runner_manager/manager/runner_manager.py Outdated
Comment thread github-runner-manager/tests/unit/factories/runner_instance_factory.py Outdated
Comment thread github-runner-manager/tests/unit/manager/test_runner_manager.py Outdated
Cleanup decisions only considered platform health and VM age, never the
cloud VM state. A VM reported as ERROR by OpenStack never recovers, yet it
kept occupying a runner slot for the full creation timeout (~23 minutes)
before being collected, and no replacement was created in the meantime.

Treat runners whose cloud VM is in ERROR state as eligible for cleanup
regardless of age, freeing the slot for a replacement on the next reconcile.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

@cbartz cbartz marked this pull request as ready for review June 26, 2026 09:22
… python 3.12

The unit job ran on a self-hosted runner whose image now provides Python
3.14. With the open-ended requires-python (>=3.10), setup-python/uv resolves
to the newest interpreter, and Python 3.14 changed the default multiprocessing
start method on Linux from fork to forkserver. forkserver pickles the Process
target, which test_flush_runner_locked cannot do (FlaskClient.post references
an unpicklable Flask init lambda), so the unit job failed.

Switch the job to GitHub-hosted ubuntu-24.04 (Noble) and pin Python to 3.12 so
tox uses a fork-default interpreter. The integration job stays on self-hosted
runners as it requires OpenStack.
@cbartz cbartz enabled auto-merge (squash) June 26, 2026 10:15
@cbartz cbartz merged commit 73a21a0 into main Jun 29, 2026
218 of 242 checks passed
@cbartz cbartz deleted the fix/cleanup-errored-vms branch June 29, 2026 13:16
@cbartz

cbartz commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator Author

Force-merged (admin) despite the failing Runner tests across bases (22.04) / test_charm_runner check.

The failure is a pre-existing flaky test unrelated to this PR:

  • The failing test is test_otel_collector_endpoint_pre_job_installs_config. It dispatches a workflow, waits for it to complete, then SSHes into the runner — but ephemeral runner VMs are torn down on job completion, so the inspection races the runner-manager cleanup. On 22.04 the cleanup won (get_single_runner found zero VMs → found more than one runners or no runners: []); the identical test passed on the 24.04 base in the same run.
  • This PR's code path was never exercised: there were zero VMs in VMState.ERROR across all attempts, so the new errored-VM cleanup logic never ran. The VMs were always ACTIVE and removed via the standard ephemeral-completion cleanup that already exists on main.

Follow-up to de-flake the test: #803.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants