Summary
Every image the engine builds locally is tagged with an unqualified name — dc-go-compose-base-<id>:latest, dc-go-final-<id>:latest. When a subsequent build uses one as its base, BuildKit resolves that reference against a registry before falling back to the local image store. The unqualified name normalizes to docker.io/library/dc-go-compose-base-<id>:latest, which of course does not exist upstream and never will.
It is non-fatal — the local store answers and the build proceeds — but it costs a pointless registry round-trip per build, and it emits errors that read exactly like a genuine registry failure.
Observed
From a compose-source devcontainer with three features, in an environment where docker.io is mirrored to a private pull-through cache:
22:23:03 devcontainer: build start (source=features, ref=dc-go-final-<id>:latest)
22:23:04 INFO trying next host after status: 403 Forbidden
dockerd.span="resolving docker.io/library/dc-go-compose-base-<id>:latest"
22:23:05 INFO fetch failed after status: 403 Forbidden
dockerd.span="resolving docker.io/library/dc-go-compose-base-<id>:latest"
22:24:55 devcontainer: build completed (source=features, image=edf56646e0e3)
22:24:56 INFO trying next host after status: 403 Forbidden
dockerd.span="resolving docker.io/library/dc-go-final-<id>:latest"
Two distinct occurrences per build:
layerFeatures builds with FROM $_DEV_CONTAINERS_BASE_IMAGE, where the arg is the dc-go-compose-base-* tag from prepareComposeServiceImage (up.go:864, consumed at up.go:546). BuildKit resolves that base against the registry.- The same thing again on the
dc-go-final-* tag, from the next consumer of that image (reconcileRemoteUserUID generates FROM $_DEV_CONTAINERS_BASE_IMAGE at useruid.go:171).
The status code is environment-dependent — a plain Docker Hub setup would give 401/404 rather than 403 — but the wasted resolve happens everywhere. A mirror just makes it loud.
Why it is worth fixing
Beyond the round-trip: these tags are engine-internal and known to exist locally, so the registry lookup can never succeed and can never be useful. The failure text is indistinguishable from a real broken-registry problem, which makes it actively misleading during triage. We just spent a while chasing a genuine 403 Forbidden on a real upstream image, and this noise sat in the same logs with the same signature.
Suggested fix
For any build whose base is an engine-generated local tag, tell the builder not to consult a registry — BuildKit's resolve-mode=local (equivalently --pull=false / not setting PullParent) on the BuildSpec in layerFeatures and reconcileRemoteUserUID.
Qualifying the tags as localhost/... would also stop the Docker Hub normalization, though that is the more invasive change of the two.
Environment
github.com/crunchloop/devcontainer v0.5.0- dockerd 29.2.1, containerd 2.3.4, containerd-snapshotter enabled
- compose-source devcontainer,
build: on the primary service, 3 features
Summary
Every image the engine builds locally is tagged with an unqualified name —
dc-go-compose-base-<id>:latest,dc-go-final-<id>:latest. When a subsequent build uses one as its base, BuildKit resolves that reference against a registry before falling back to the local image store. The unqualified name normalizes todocker.io/library/dc-go-compose-base-<id>:latest, which of course does not exist upstream and never will.It is non-fatal — the local store answers and the build proceeds — but it costs a pointless registry round-trip per build, and it emits errors that read exactly like a genuine registry failure.
Observed
From a compose-source devcontainer with three features, in an environment where
docker.iois mirrored to a private pull-through cache:Two distinct occurrences per build:
layerFeaturesbuilds withFROM $_DEV_CONTAINERS_BASE_IMAGE, where the arg is thedc-go-compose-base-*tag fromprepareComposeServiceImage(up.go:864, consumed atup.go:546). BuildKit resolves that base against the registry.dc-go-final-*tag, from the next consumer of that image (reconcileRemoteUserUIDgeneratesFROM $_DEV_CONTAINERS_BASE_IMAGEatuseruid.go:171).The status code is environment-dependent — a plain Docker Hub setup would give 401/404 rather than 403 — but the wasted resolve happens everywhere. A mirror just makes it loud.
Why it is worth fixing
Beyond the round-trip: these tags are engine-internal and known to exist locally, so the registry lookup can never succeed and can never be useful. The failure text is indistinguishable from a real broken-registry problem, which makes it actively misleading during triage. We just spent a while chasing a genuine
403 Forbiddenon a real upstream image, and this noise sat in the same logs with the same signature.Suggested fix
For any build whose base is an engine-generated local tag, tell the builder not to consult a registry — BuildKit's
resolve-mode=local(equivalently--pull=false/ not settingPullParent) on theBuildSpecinlayerFeaturesandreconcileRemoteUserUID.Qualifying the tags as
localhost/...would also stop the Docker Hub normalization, though that is the more invasive change of the two.Environment
github.com/crunchloop/devcontainerv0.5.0build:on the primary service, 3 features