Skip to content

kubernetes: wait for exec connection readiness - #4077

Open
jarqvi wants to merge 2 commits into
docker:masterfrom
jarqvi:fix-kubernetes-first-connection-race
Open

jarqvi wants to merge 2 commits into
docker:masterfrom
jarqvi:fix-kubernetes-first-connection-race

Conversation

@jarqvi

@jarqvi jarqvi commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

fixes #4040

The kubernetes driver dials a builder pod by running buildctl dial-stdio
through the exec API. ExecConn started StreamWithContext in a goroutine and
returned the connection immediately, before the SPDY stream was up. If the
upgrade failed (e.g. tls: internal error on a fresh node whose kubelet cert
isn't signed yet), Dial() had already reported success, so tryWithBackoff
never retried and the error surfaced later as a gRPC client-preface error.

Wait until the stream is established before returning. The executor reads stdin
only once its streams are created, so the first stdin read is used as the
readiness signal — no sleeps or probes. Transient setup failures are now
returned synchronously so tryWithBackoff can retry, and context cancellation
still unblocks the wait.

Added unit tests for newExecConn and tryWithBackoff using a fake executor.

Signed-off-by: MohammadHasan Akbari <jarqvi.jarqvi@gmail.com>
Comment thread driver/kubernetes/execconn/execconn_test.go
@crazy-max crazy-max added this to the v0.38.0 milestone Sep 14, 2026
Signed-off-by: MohammadHasan Akbari <jarqvi.jarqvi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

driver/kubernetes: same first-connection race as #4021 — Dial() returns before SPDY stream is ready

2 participants