Give the manager its own trusted bundle for the Tigera OIDC type

[tmjd]

Description

Bug fix. On clusters configured with the Tigera OIDC type, calico-manager and es-calico-kube-controllers repeatedly roll their pods, alternating between two pod templates.

Impact

This is not a steady loop. Neither controller rewrites the bundle on every reconcile — the 5 minute periodic resync does not re-trigger it — so rollouts arrive in bursts, each set off by something that makes one of the controllers rebuild the bundle from cold: an upgrade, a manager restart, or a certificate rotation. Within a burst, replacements follow one another within seconds; between bursts the deployments can sit quiet for hours.

Both deployments stay available the whole time, since each flip is an ordinary rolling update. The cost is repeated unnecessary restarts, log and event noise, write contention on the shared ConfigMap (Operation cannot be fulfilled on configmaps "tigera-ca-bundle"), and a revision history that grows without bound — not downtime.

Root cause

For Authentication.spec.oidc.type: Tigera, handleCloudReconcile discards the manager's component-named trusted bundle and replaces it with one from CreateTrustedBundleWithSystemRootCertificates(), which renders the default-named tigera-ca-bundle (manager_controller_cloud.go:80-87):

if authenticationCR != nil && authenticationCR.Spec.OIDC != nil && authenticationCR.Spec.OIDC.Type == operatorv1.OIDCTypeTigera {
	bundleMaker, err = certificateManager.CreateTrustedBundleWithSystemRootCertificates()
}

That ConfigMap is owned by the core controller, which renders it with a different set of certificates — the node and typha key pairs, plus the legacy tigera-operator/typha-ca ConfigMap on clusters old enough to still carry one (core_controller.go:1978-1990). Both land in the same namespace, since render.ManagerNamespace is common.CalicoNamespace.

Two controllers writing one ConfigMap with different contents overwrite each other, so calico-system/tigera-ca-bundle alternates between exactly two values. Both consumers flip with it:

• calico-manager mounts the bundle and hashes its data keys, so its pod-template annotations flip.
• es-calico-kube-controllers obtains it via cm.LoadTrustedBundle(), which copies the ConfigMap's hash.operator.tigera.io/* annotations onto the dependent workload — including the namespace-prefixed calico-system.hash.operator.tigera.io/typha-ca key, which appears and disappears.

Each deployment therefore oscillates between two pod templates. Kubernetes reuses the two ReplicaSets rather than creating new ones and simply bumps deployment.kubernetes.io/revision on each switch back, so a deployment can reach a revision number in the hundreds or thousands while only ever having created a handful of ReplicaSets. Diffing the two ReplicaSets of either deployment shows the typha-ca annotation and the bundle content hash as the only pod-template differences.

Which clusters this affects

The extra certificate comes from the legacy typha-ca ConfigMap, which only exists on clusters installed by an operator older than v1.29: b0fce5783 (2022-03-04) replaced the dedicated Typha/Felix CA with the central CA, and nothing has ever deleted the old ConfigMap. Where one is present the two writers' bundles differ and the deployments flip. On a cluster installed after that change neither bundle contains it, both writers produce identical content, the second write is a no-op, and nothing flaps.

Deleting the stale ConfigMap therefore stops the churn, but that is a mitigation rather than a fix — the two-writer conflict remains latent and resurfaces wherever the two controllers' inputs diverge for any other reason.

The change

The only thing the Tigera OIDC type actually needs is the system root certificates, so pass that as a flag to the one call that already builds the manager's bundle, rather than rebuilding it:

includeSystemRoots := authenticationCR != nil && authenticationCR.Spec.OIDC != nil &&
	authenticationCR.Spec.OIDC.Type == operatorv1.OIDCTypeTigera

bundleMaker, err := certificateManager.CreateNamedTrustedBundleFromSecrets(TrustedBundlePrefix, r.client,
	helper.TruthNamespace(), includeSystemRoots, trustedSecretNames...)

authenticationCR is already fetched above this point, so the decision can be made here. The manager is then always backed by a bundle named for itself, and handleCloudReconcile no longer takes part in building it — it only checks that the certificates the bundle trusts are available, and its authenticationCR parameter is gone.

That also removes a redundant AddCertificates loop. AddCertificates appends rather than deduplicating by name (it skips only nil entries and certificates signed by our own CA), so on the cloud path every trusted secret was added a second time and any certificate not signed by our CA — a BYO cert — ended up duplicated in the bundle. Certificates are now added exactly once, by the single constructor call.

Upgrade impact

The manager now creates and mounts calico-manager-ca-bundle-system-certs instead of the shared tigera-ca-bundle. Worth a reviewer's attention:

• Nothing is orphaned — the core controller continues to own and maintain tigera-ca-bundle for its other consumers.
• In-container certificate paths are unchanged: VolumeMounts() uses the constant TrustedCertVolumeMountPath, and only the ConfigMap and volume names derive from the bundle name. Voltron and manager keep reading the same files.
• calico-manager rolls once on upgrade as the new volume is mounted.

Testing

• New unit test should leave the manager's trusted bundle alone on the cloud path passes a component-named bundle into handleCloudReconcile and asserts the bundle handed back is still that one, and is not tigera-ca-bundle. Verified it fails if the override is reintroduced, producing tigera-ca-bundle.
• The test drives handleCloudReconcile rather than Reconcile, because reaching this code through Reconcile requires utils.GetKeyValidatorConfig → tigerakvc.New, which performs live OIDC discovery and JWKS fetches against the issuer. That is also why this path had no prior unit coverage at all. It does mean the includeSystemRoots expression itself is not unit-covered; a fake-issuer harness would let a future test cover it, and would be a worthwhile follow-up.
• go test ./pkg/controller/manager/... passes (29 specs).
• gofmt clean; the repo pre-commit hook passes.

Components affected

The manager controller on the Tigera OIDC path. Multi-tenant and other OIDC types already used the named bundle and are unaffected.

Release Note

Fixed calico-manager and es-calico-kube-controllers repeatedly rolling their pods, caused by the manager controller and the core controller writing the shared tigera-ca-bundle ConfigMap with different sets of certificates.

For PR author

• Tests for change.
• If changing pkg/apis/, run make gen-files
• If changing versions, run make gen-versions

For PR reviewers

A note for code reviewers - all pull requests must have the following:

• Milestone set according to targeted release.
• Appropriate labels:
  • kind/bug if this is a bugfix.
  • kind/enhancement if this is a a new feature.
  • enterprise if this PR applies to Calico Enterprise only.