HDDS-16504. Datanode startup can hang indefinitely when Ratis group recovery stalls - #11264
Conversation
…ecovery stalls (add a timeout/watchdog around container initialization) - Initial Commit
adoroszlai
changed the title
| } catch (TimeoutException e) { | ||
| // Best-effort interrupt; a thread blocked on disk I/O may not respond, | ||
| // but the datanode will shut down once the caller marks startup FAILED. | ||
| initFuture.cancel(true); | ||
| throw new IOException("OzoneContainer initialization did not complete within " + initTimeout | ||
| + ". Failing datanode startup; a stalled Ratis group recovery or volume I/O is the likely cause.", e); |
There was a problem hiding this comment.
Could we prevent initialization from continuing after the timeout, and coordinate cleanup with the surviving initializer?
cancel(true) and shutdownNow() only interrupt the worker. Ratis 3.2.1's RaftServerProxy.startImpl() waits on CompletableFuture.join(), which ignores interruption. If recovery completes later, initializeContainerServices() continues starting services even though startup is already FAILED and shutdown may have cleaned up their dependencies.
There was a problem hiding this comment.
Good catch, thanks — you're right, and I confirmed it. cancel(true)/shutdownNow() only interrupt the worker, and the interrupt is ignored by the stalled Ratis RaftServerProxy.startImpl() CompletableFuture.join(). I have handled in different way using separate watch thread. Kindly have a re-look.
…o timeout and terminate the datanode
…o timeout and terminate the datanode
…o timeout and terminate the datanode
smengcl
left a comment
smengcl
left a comment
There was a problem hiding this comment.
looks good now. Thanks @devmadhuu
|
Thanks @devmadhuu for the patch. |
3 participants
What changes were proposed in this pull request?
Datanode startup can hang indefinitely when container initialization stalls without throwing — typically when the Ratis write channel is recovering Raft groups and a read blocks on a failing volume. OzoneContainer.start() never returns, so the datanode never registers/heartbeats, logs endpoint‑task timeouts forever, and re‑hits the same stall on restart.
This complements HDDS‑16425 (which handles init failures that throw); this covers the hang case.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16504
How was this patch tested?