GPU fault classification: cover the child pods of distributed tasks - #7913
Draft
samhita-alla wants to merge 2 commits into
Draft
GPU fault classification: cover the child pods of distributed tasks#7913samhita-alla wants to merge 2 commits into
samhita-alla wants to merge 2 commits into
Conversation
|
This PR was flagged by our automated quality checks. If you're a genuine We appreciate your contribution and apologize if this is a false positive! |
samhita-alla
marked this pull request as draft
August 25, 2026 13:11
samhita-alla
force-pushed
the
gpu-forward/gpu-failure-classification
branch
from
August 26, 2026 06:43
c56770a to
54e3a5e
Compare
This was referenced Aug 26, 2026
samhita-alla
force-pushed
the
gpu-forward/gpu-failure-classification
branch
2 times, most recently
from
August 26, 2026 06:59
3176487 to
f4abf56
Compare
samhita-alla
force-pushed
the
gpu-forward/child-pod-gpu-fault
branch
from
August 26, 2026 13:08
8ef1440 to
e9767e0
Compare
Base automatically changed from
gpu-forward/gpu-failure-classification
to
main
August 27, 2026 05:30
Classification so far covered tasks whose tracked resource is the Pod itself. A RayJob, PyTorchJob, MPIJob or TFJob records the Xid on a worker pod while the executor tracks the CRD, so the fault the node saw never reached the failure the user read, and a GPU that fell off the bus during distributed training still looked like the workload's own crash. Nothing new has to be watched to fix that. The event informer is not scoped to the tracked resource, so a worker pod's events are already cached, and the Pod informer is already running and already selects on the managed label that every plugin merges into the pod templates it builds. What was missing was a way to name the pods: an operator's child pod names carry a random suffix, so the executor cannot derive the keys the events are cached under. ChildPodDiscovery is the optional interface that closes that, in the shape PluginAbortOverride already established. A plugin returns the label selector that picks out this attempt's pods; the manager lists them from the resource's own namespace out of the cache it already holds, and reads each one's faults. Because the pods arrive as objects, their UIDs are known, so the identity check that already guarded the Pod path applies unchanged and never has to fall back to matching on a name. AttemptPodSelector builds the half of that selector which holds whichever operator created the pod. The run, action and attempt labels move to flytek8s next to the managed label, for the reason that one already lives there: the plugins that build pod templates have to keep them intact. It refuses to build a selector when any of them is missing, since run and action are dropped when sanitization empties them and selecting on what is left would reach another action's pods. PreservedPodLabels exposes the same key set for plugins to re-apply after any labels a user supplied, so that the two halves cannot disagree about which labels have to survive. Which pods are consulted is narrower than everything the selector matches. A worker that finished its work cannot be why the job failed, so a succeeded pod contributes nothing even if it logged a fault mid-run; crediting that to a launcher's user-code failure would turn a user error into a hardware one and stop charging the retry. A pod still running is kept, because a worker wedged on a GPU that fell off the bus is exactly the case this path exists for. Relevance is anchored per pod on that pod's own container termination, because every CRD plugin stamps the failure with the time of the reconcile that noticed it rather than with anything the kubelet recorded, and anchoring on the reconcile would age out a fault that killed a worker minutes earlier. Init containers are not eligible anchors: they finish before the workload starts, and a native sidecar declared among them is reaped after everything else. The anchor is never later than the attempt's own failure, so a pod torn down long afterwards cannot let faults recorded in the meantime explain it. Which fault wins needs no new rule: ClassifyFailure already takes the worst severity and the earliest within it, so the faults gathered across pods are only sorted back into order before being handed over. That order is on when each fault was first recorded rather than when it was last seen, because a fault that is still repeating has a later last observation than a one-shot fault that followed it, and ordering on that would let a downstream symptom outrank the root cause. The last observation still decides relevance, which is all it was ever for. The one pod whose fault settled the verdict is named as an additional reason, because the fault sentence carries the Xid, the GPU and the node but not the pod, which on a job with sixty-four workers is not enough to act on. Only that pod is named: pointing at every pod that saw any fault would put a misleading reason on a failure the faults did not explain, and would emit one cluster event per pod when a whole node's worth of GPUs faults at once. The single pod path names nothing, since the failure is already reported against that pod. A gang-scheduled restart creates fresh pods under the same attempt and its faults count, while an earlier Flyte attempt's pods are excluded by the attempt label. A plugin that does not implement the interface contributes no pods and so no faults, which is exactly what it did before. A pod already torn down with its job is not recovered. The events are cached under a name nothing can reconstruct, so discovery logs at debug and classifies nothing rather than guessing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
… task The Ray, Kubeflow and clustered plugins track a CRD, so they implement ChildPodDiscovery to tell the framework where the pods that actually ran the task are. All three build the same selector: the attempt's own labels, which the pod templates they hand the operator already carry, narrowed by one label the operator applies to the pods themselves. For Kubeflow that is training.kubeflow.org/job-name, which is the CR's own name and so is known from the moment the resource exists; one implementation in common covers PyTorchJob, MPIJob and TFJob, because all three get their replica pods from the same operator. For the clustered plugin it is jobset.sigs.k8s.io/jobset-name, exported by the vendored JobSet API the plugin already imports. For Ray it is ray.io/cluster, which KubeRay stamps on head and worker pods and overwrites on the template, and whose value is only knowable from the RayJob's status because KubeRay appends a random suffix to the cluster name. Until that status is reported Ray falls back to the attempt labels alone. That is not a wider search in any sense that matters, since run, action and attempt already pin the selector to one attempt of one action; the most it can add is the job submitter pod, which has no GPU and so no faults to contribute. The alternative, declining until the operator catches up, would lose the fault on exactly the runs where the cluster never came up healthy. Ray also has to stop a task from taking its own pods out of reach. The head and worker templates let a task supply k8s_pod metadata, and that metadata was unioned after the execution labels with only the managed label re-forced afterwards, so a user label named run, action or attempt would overwrite the identity the selector looks a pod up by. All three templates now re-apply flytek8s.PreservedPodLabels last, which is the same key set the selector is built from. Dask and Spark are left for a follow-up. The conformance tests are the point of the test additions. Each asserts that the labels the plugin puts on the pod templates satisfy the selector the same plugin hands the framework, for every replica type it builds. The two halves live far apart and drift on either one would leave a GPU fault on a worker silently unclassified, which is the failure mode this whole path exists to prevent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
samhita-alla
force-pushed
the
gpu-forward/child-pod-gpu-fault
branch
from
August 27, 2026 06:05
e9767e0 to
23a0ece
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#7879 classifies GPU faults into the failure the user sees, but only for tasks whose tracked resource is the Pod. CRD-backed distributed tasks — RayJob, PyTorchJob, MPIJob, TFJob, and the clustered JobSet plugin — track the CRD while the fault event lands on a worker pod the plugin manager never looks at, so an Xid on a training worker still surfaces as a generic operator failure. This closes that gap, which #7879's description tracks as its known follow-up.
How
Nothing new is watched. The executor's event cache already holds the Xid events recorded on operator-created pods, and every plugin already merges the framework's run, action and attempt labels into the pod templates it builds. What was missing is the lookup key from a failing CRD to its pods.
k8s.ChildPodDiscovery, an optional interface next toPluginAbortOverrideand in the same shape: a plugin whose tracked resource is not a Pod can return the label selector that names the pods of this attempt. A plugin that does not implement it behaves exactly as before.flytek8s.AttemptPodSelectorbuilds the shared half of that selector from the labels the framework stamps on every task (managed, run, action, attempt), and returns nil when the attempt cannot be identified, because a partial match would credit one action's fault to another. The label keys move from the executor into flytek8s so the stamping side and the selecting side cannot disagree;flytek8s.PreservedPodLabelsre-forces the same keys last in template builders so a user-supplied pod label namedrunorattemptcannot break the join.ray.io/clusterfrom the RayJob status (falling back to the attempt selector alone before the operator reports the cluster name), the three Kubeflow plugins sharetraining.kubeflow.org/job-name, and clustered usesjobset.sigs.k8s.io/jobset-name. Every value is verified against the vendored operator, and a conformance test per plugin asserts the pod templates the plugin itself builds satisfy its own selector, so label drift on either side fails a unit test.Faults on a child pod that was deleted before the failing reconcile are lost rather than guessed at; discovery logs at debug when that happens. Dask and Spark are deliberately left out of this PR; both are mechanical follow-ups on the same interface.
Tests
go build ./...in both modules; 38 packages tested across flyteplugins and the executor.golangci-lint(v2.12.0) diffed against the base on both configs: no new findings.AttemptPodSelectorlabel cases; per-pluginChildPodsbehavior including Ray's empty cluster name; per-plugin conformance of built pod templates against the plugin's own selector across every replica type; plugin-manager cases for earliest-critical-wins across workers, aggregated versus one-shot event ordering, succeeded workers excluded, still-running wedged workers included, different-attempt pods excluded, wrong UID rejected, vanished pods contributing nothing, user label overrides not breaking the selector, and non-implementing plugins unchanged.Stack: #7878 → #7879 (base) → this.
🤖 Generated with Claude Code
main