Test: add E2E configuration matrix workflow for TWC by niyomukiza-mechack · Pull Request #582 · temporalio/temporal-worker-controller

niyomukiza-mechack · 2026-09-09T21:07:33Z

What was changed

Added a new E2E configuration matrix workflow that deploys TWC to a Kind cluster with different Helm value configurations and verifies the controller actually starts and works.

Currently tests three modes: default (cluster-wide with cert-manager), namespace-scoped (restrictWatchNamespaces), and BYO cert (certmanager.enabled=false with a custom certSecretName).
Each mode verifies that controller pods reach Running state, there are no error-level log entries, and the webhook accepts a test WorkerDeployment.

This would have caught both the namespace-scoped ClusterConnection watch bug and the hard-coded cert Secret name issue before they shipped.

The goal is to expand the matrix over time as we discover new customer deployment configurations - each new mode is one entry in the matrix, so adding coverage is lightweight.

Why?

Checklist

Related to #584

Closes
How was this tested:

Any docs updates needed?

github-actions · 2026-09-09T21:08:26Z

+    name: e2e-${{ matrix.mode.name }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    strategy:
+      fail-fast: false
+      matrix:
+        mode:
+          - name: default
+            helm-args: "--set certmanager.enabled=true"
+            install-cert-manager: true
+          - name: namespace-scoped
+            helm-args: "--set rbac.restrictWatchNamespaces='{default}' --set certmanager.enabled=true"
+            install-cert-manager: true
+          - name: byo-cert
+            helm-args: "--set certmanager.enabled=false --set webhook.certSecretName=byo-webhook-cert"
+            install-cert-manager: false
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Create Kind cluster
+        uses: helm/kind-action@v1
+        with:
+          cluster_name: twc-e2e
+
+      - name: Build controller image
+        run: docker build -t temporal-worker-controller:e2e .
+
+      - name: Load image into Kind
+        run: kind load docker-image temporal-worker-controller:e2e --name twc-e2e
+
+      - name: Install cert-manager
+        if: matrix.mode.install-cert-manager
+        run: |
+          helm install cert-manager jetstack/cert-manager \
+            --namespace cert-manager \
+            --create-namespace \
+            --set crds.enabled=true \
+            --wait --timeout 2m
+
+      - name: Generate BYO certificate
+        if: ${{ !matrix.mode.install-cert-manager }}
+        run: |
+          kubectl create namespace temporal-system --dry-run=client -o yaml | kubectl apply -f -
+          openssl req -x509 -newkey rsa:2048 -keyout tls.key -out tls.crt \
+            -days 1 -nodes -subj "/CN=temporal-worker-controller-webhook-service.temporal-system.svc"
+          kubectl create secret tls byo-webhook-cert \
+            --cert=tls.crt --key=tls.key -n temporal-system
+
+      - name: Install CRDs chart
+        run: |
+          helm install temporal-worker-controller-crds helm/temporal-worker-controller-crds \
+            --namespace temporal-system \
+            --create-namespace
+
+      - name: Install TWC
+        run: |
+          helm install temporal-worker-controller helm/temporal-worker-controller \
+            --namespace temporal-system \
+            --set image.repository=temporal-worker-controller \
+            --set image.tag=e2e \
+            --set image.pullPolicy=IfNotPresent \
+            ${{ matrix.mode.helm-args }}
+
+      - name: Wait for controller pods
+        run: |
+          kubectl wait --for=condition=ready pod \
+            -l app.kubernetes.io/name=temporal-worker-controller \
+            -n temporal-system --timeout=180s
+
+      - name: Check for error logs
+        run: |
+          errors=$(kubectl logs -n temporal-system \
+            -l app.kubernetes.io/name=temporal-worker-controller \
+            --tail=100 2>&1 | grep '"level":"error"' || true)
+          if [ -n "$errors" ]; then
+            echo "ERROR: Found error-level logs:"
+            echo "$errors"
+            exit 1
+          fi
+          echo "No error logs detected"
+
+      - name: Apply test WorkerDeployment
+        run: kubectl apply -f test/e2e/fixtures/basic-wd.yaml
+
+      - name: Verify WorkerDeployment accepted
+        run: |
+          kubectl get workerdeployment test-worker -n temporal-system
+          echo "WorkerDeployment accepted by webhook"
+
+
+


Opengrep — security.gha.missing-explicit-permissions (ERROR)

No explicit GITHUB_TOKEN permissions found at the workflow or job level. Add a permissions: block at the workflow root (applies to all jobs) or per job with least privilege (e.g., contents: read and only specific writes like pull-requests: write if needed).

Fixed in 0d6765d

Fixed in 6a0a82a

Fixed in fa1ba4e

Fixed in 8aedcf0

Fixed in 85bb3cf

Fixed in 2b99f7e

Fixed in 5d8e64d

Fixed in 1c92e5c

Fixed in 8362870

Fixed in a43b0e3

Fixed in dd61724

Fixed in 1222f22

Fixed in cf3240d

github-actions · 2026-09-09T21:08:26Z

+            install-cert-manager: false
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4


Opengrep — security.gha.unpinned-action (WARNING)

Unpinned action reference actions/checkout@v4: this uses: resolves a mutable ref (tag or branch), so the code that runs in CI can change without this line changing. A compromised upstream can repoint the tag and execute arbitrary code with access to this repository's secrets and GITHUB_TOKEN (tj-actions/changed-files, March 2025). Pin to the full 40-character commit SHA with the resolved version in a trailing comment, e.g. uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2. Prefer deputy pin --ecosystems github-actions, which resolves the ref, writes a version comment that reflects the most specific ref actually pointing at that commit, and verifies the SHA is reachable from a real branch upstream. That last check matters: pinning alone does not detect imposter or dangling commits, and this rule only sees the shape of the ref, never its provenance. Reusable workflow calls (owner/repo/.github/workflows/x.yml@ref) run with the same trust as actions and are pinned the same way. Not reported, by campaign policy: temporalio/* refs (first-party, pinned by internal process), local ./ actions, self-repository $/ refs (resolve to the running commit, so they are already pin-equivalent), and docker:// images (pinned by digest as a separate ecosystem).

Fixed in 0d6765d

Fixed in 6a0a82a

Fixed in fa1ba4e

Fixed in 8aedcf0

Fixed in 85bb3cf

Fixed in 2b99f7e

Fixed in 5d8e64d

Fixed in 1c92e5c

Fixed in 8362870

Fixed in a43b0e3

Fixed in dd61724

Fixed in 1222f22

Fixed in cf3240d

github-actions · 2026-09-09T21:08:27Z

+        uses: actions/checkout@v4
+
+      - name: Create Kind cluster
+        uses: helm/kind-action@v1


Opengrep — security.gha.unpinned-action (WARNING)

Unpinned action reference helm/kind-action@v1: this uses: resolves a mutable ref (tag or branch), so the code that runs in CI can change without this line changing. A compromised upstream can repoint the tag and execute arbitrary code with access to this repository's secrets and GITHUB_TOKEN (tj-actions/changed-files, March 2025). Pin to the full 40-character commit SHA with the resolved version in a trailing comment, e.g. uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2. Prefer deputy pin --ecosystems github-actions, which resolves the ref, writes a version comment that reflects the most specific ref actually pointing at that commit, and verifies the SHA is reachable from a real branch upstream. That last check matters: pinning alone does not detect imposter or dangling commits, and this rule only sees the shape of the ref, never its provenance. Reusable workflow calls (owner/repo/.github/workflows/x.yml@ref) run with the same trust as actions and are pinned the same way. Not reported, by campaign policy: temporalio/* refs (first-party, pinned by internal process), local ./ actions, self-repository $/ refs (resolve to the running commit, so they are already pin-equivalent), and docker:// images (pinned by digest as a separate ecosystem).

Fixed in 0d6765d

Fixed in 6a0a82a

Fixed in fa1ba4e

Fixed in 8aedcf0

Fixed in 85bb3cf

Fixed in 2b99f7e

Fixed in 5d8e64d

Fixed in 1c92e5c

Fixed in 8362870

Fixed in a43b0e3

Fixed in dd61724

Fixed in 1222f22

Fixed in cf3240d

zainawaisn · 2026-09-15T00:05:51Z

+      fail-fast: false
+      matrix:
+        mode:
+          - name: default


webhook.enabled is set to false by default in values.yaml for WorkerDeployment webhooks.

You can either add a "--set webhook.enabled=true" arg to each individual helm-arg or add "--set webhook.enabled=true" to the shared name: Install TWC step.

- name: Install TWC run: | helm install temporal-worker-controller helm/temporal-worker-controller \ --namespace temporal-system \ --set image.repository=temporal-worker-controller \ --set image.tag=e2e \ --set image.pullPolicy=IfNotPresent \ --set webhook.enabled=true \ ${{ matrix.mode.helm-args }}

As a follow-up to this, the byo-cert mode does not set certmanager.caBundle. When the webhooks are enabled and cert-manager.installed=false, byo-cert will need to assign the cert from "Generate BYO certificate" step to certmanager.caBundle in order to validate the self-signed cert.

I agree but this was done on purpose. Getting this workflow green in CI took about 10 rounds of fixes due to the subchart dependency still being in Chart.yaml. Every change that touched the cert setup or install order broke something new, so I simplified things to get to green. No webhook.enabled=true, no caBundle, no custom cert Secret name. The plan was/is once the subchart removal #575 merges, I'll rebase and enable everything. The webhook.enabled=true, caBundle for BYO, custom certSecretName, and remove the dependency build step. Sorry I should have mentioned this in the PR description

No worries, this sounds good then. These items can be done as a follow-up after #575.

…s it

…rges

… merged

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

niyomukiza-mechack requested review from a team, eniko-dif and jlegrone as code owners September 9, 2026 21:07

niyomukiza-mechack requested review from jaypipes and tomba7 September 9, 2026 21:08

github-actions Bot reviewed Sep 9, 2026

View reviewed changes

niyomukiza-mechack mentioned this pull request Sep 10, 2026

[Feature Request] Add E2E configuration matrix tests for TWC #584

Open

zainawaisn reviewed Sep 15, 2026

View reviewed changes

zainawaisn approved these changes Sep 15, 2026

View reviewed changes

niyomukiza-mechack added 12 commits September 15, 2026 10:12

Test: add E2E configuration matrix workflow for TWC

25ad4b6

fix: add Helm repo and disable cert-manager subchart in E2E

06a8ba6

fix: build Helm dependencies before install in E2E

ef3ebe1

fix: wait for webhook cert Secret before checking pods

a730c73

fix: wait for cert Secret before restarting pods in E2E

b0e13f4

fix: quote variable to satisfy shellcheck in E2E workflow

d15a93d

fix: handle BYO cert Secret name in wait step

bdcbc26

fix: remove redundant kubectl wait step, rollout status already cover…

1a03e6a

…s it

fix: add diagnostics step on failure for E2E debugging

4316ed6

fix: use default cert Secret name in BYO mode until certSecretName me…

3a58eda

…rges

fix: wait step checks correct Secret name for all modes

7719b22

refactor: enable full E2E testing now that subchart removal (#575) is…

dd61724

… merged

niyomukiza-mechack force-pushed the test/e2e-config branch from a43b0e3 to dd61724 Compare September 15, 2026 17:21

niyomukiza-mechack added this to the vNextNext milestone Sep 15, 2026

niyomukiza-mechack and others added 2 commits September 15, 2026 10:42

fix: add helm repo and SAN to BYO cert for webhook TLS

1222f22

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix: shellcheck disable for EXTRA_ARGS, add webhook readiness wait

cf3240d

niyomukiza-mechack removed this from the vNext milestone Sep 16, 2026

Navigation Menu

Test: add E2E configuration matrix workflow for TWC - #582

Test: add E2E configuration matrix workflow for TWC#582
niyomukiza-mechack wants to merge 14 commits into
mainfrom
test/e2e-config

niyomukiza-mechack commented Sep 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot Sep 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot Sep 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot Sep 9, 2026 •

edited

Loading

Uh oh!

zainawaisn Sep 15, 2026

Uh oh!

zainawaisn Sep 15, 2026 •

edited

Loading

Uh oh!

niyomukiza-mechack Sep 15, 2026

Uh oh!

zainawaisn Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

niyomukiza-mechack commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why?

Checklist

Uh oh!

github-actions Bot Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zainawaisn Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

zainawaisn Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

niyomukiza-mechack Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

zainawaisn Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

niyomukiza-mechack commented Sep 9, 2026 •

edited

Loading

github-actions Bot Sep 9, 2026 •

edited

Loading

github-actions Bot Sep 9, 2026 •

edited

Loading

github-actions Bot Sep 9, 2026 •

edited

Loading

zainawaisn Sep 15, 2026 •

edited

Loading