Skip to content

controller: mark nodes stuck rebooting as degraded - #159

Merged
alicefr merged 1 commit into
bootc-dev:mainfrom
ptalgulk01:reboot-timeout-degraded
Sep 7, 2026
Merged

controller: mark nodes stuck rebooting as degraded#159
alicefr merged 1 commit into
bootc-dev:mainfrom
ptalgulk01:reboot-timeout-degraded

Conversation

@ptalgulk01

@ptalgulk01 ptalgulk01 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

What

Adds spec.rollout.rebootTimeoutSeconds to BootcNodePool. If a node stays in
the Rebooting state longer than this (e.g. it failed to boot the new image or
never rejoined the cluster), the controller reports it as degraded at the pool
level (degradedCount + Degraded/NodeDegraded). If 2+ nodes time out, the
rollout halts (Degraded/RolloutHalted).

The field is optional; when unset the controller waits indefinitely, so existing
behavior is unchanged.

Why

Today a node that reboots and never comes back sits in Rebooting forever. The
daemon can't report it (the node is down), so nothing surfaces the failure and
the reboot slot is held indefinitely. This covers issue #69 item 4 and the
"cross-distro bad image" scenario.
Fixes: #158

How it works

  • driveRollout reclassifies timed-out Rebooting nodes into the degraded
    bucket and requeues the pool so the timeout fires without needing an event.
  • findUnhealthySlots counts timed-out nodes toward the halt threshold — a node
    that failed to boot the target is a strong signal the image is bad, which is
    what the halt is meant to catch.

Design note

The controller reports at the pool level only. It has read-only access to
bootcnodes/status, and the daemon owns that condition (it auto-clears Degraded
on recovery). Marking the BootcNode's own condition would need an RBAC change and
a second writer for daemon-owned status. Happy to go that route instead if
maintainers prefer — flagging it here for input.

Testing

Unit tests for the timeout decision and slot classification, plus envtests for
the pool-degraded and rollout-halt paths.
Assisted-by: AI

Add rollout.rebootTimeoutSeconds so a node that stays in Rebooting past
the timeout is reported degraded at the pool level, and 2+ such nodes
halt the rollout.

Assisted-by: AI
Signed-off-by: Prachiti Talgulkar <ptalgulk01@users.noreply.github.com>
@ptalgulk01
ptalgulk01 force-pushed the reboot-timeout-degraded branch from ec9059d to 7040478 Compare September 7, 2026 11:36
@alicefr

alicefr commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

It looks great, many thanks for this work! Nice coverage also with the env tests

@alicefr
alicefr merged commit 0e0bac3 into bootc-dev:main Sep 7, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Detect nodes that never recover after reboot (reboot timeout → degraded)

2 participants