From 7d15493a3d9cffb5da950720a821cef60c497f00 Mon Sep 17 00:00:00 2001
From: Jorge Garcia Oncins <jgarciao@redhat.com>
Date: Wed, 26 Aug 2026 17:23:43 +0200
Subject: [PATCH 1/2] feat(e2e): make e2e:kubernetes work transparently on
 OpenShift

Running `mise run e2e:kubernetes` on OpenShift required manual namespace
creation, SCC grants, Helm value overrides, and cleanup. A separate
`e2e:openshift` task existed but only checked pod readiness without
running the Rust e2e test suite, and even with the suite wired up the
SSH-relay `sandbox connect` path stalled to the ready timeout because
`kubectl port-forward` cannot carry round-trip-heavy SSH over the
internet.

The harness now auto-detects OpenShift via the `route.openshift.io` API
group and, on OpenShift, both configures the cluster and switches the
gateway transport automatically:

- Drives the gateway through a passthrough OpenShift Route secured with
  mandatory mTLS instead of port-forward, so the connect suites
  (live_policy_update, port_forward, sync, connect-based
  sandbox_lifecycle, settings_management) actually pass. Computes the
  Route host from the cluster ingress domain, extracts client mTLS
  material from the openshell-client-tls secret, waits for the Route to
  serve mTLS, asserts a certless caller is rejected at the TLS
  handshake, and registers an mTLS CLI gateway pointing at the Route.
- Applies an SCC-compatible Helm values overlay that removes hardcoded
  runAsUser/fsGroup, letting OpenShift assign UIDs from the namespace
  range.
- Grants the privileged SCC to openshell-sandbox before Helm install
  and removes it during cleanup.
- Grants the anyuid SCC to the PostgreSQL fixture service account in
  DB scenarios and removes it during cleanup.
- All oc commands use --context to target the correct cluster.

The OpenShift e2e overlay (ci/values-openshift-e2e.yaml) turns TLS back
on, enables the Route, promotes the cert-verified caller to a dev
principal, and forces `image.pullPolicy`/`supervisor.image.pullPolicy`
to Always so runs against the `latest` upstream image use it instead of
a stale copy cached on the cluster nodes. Every OpenShift branch is
gated on OPENSHIFT_DETECTED, so the vanilla-Kubernetes port-forward path
is unchanged.

The Helm template for podSecurityContext is wrapped with {{- with }} so
null values omit the block instead of rendering invalid YAML.

The separate e2e:openshift task and e2e-openshift.sh script are removed
since e2e:kubernetes now covers OpenShift.

TESTING.md is updated with Kubernetes e2e documentation including
OpenShift auto-detection, dropping the e2e-host-gateway feature on
remote clusters, pinning IMAGE_TAG when the CLI and image versions
differ, task variants, and environment variables.

The debug-openshell-cluster skill gains an OpenShift platform row and
two SCC failure patterns (gateway rejected over hardcoded runAsUser,
sandbox missing the privileged SCC) covering the SCC handling and
podSecurityContext behavior this change introduces.

Signed-off-by: Jorge Garcia Oncins <jgarciao@redhat.com>
---
 TESTING.md                                    | 112 ++++++
 .../openshell/ci/values-openshift-e2e.yaml    |  45 +++
 .../openshell/ci/values-openshift-scc.yaml    |  20 +
 .../openshell/templates/_gateway-workload.tpl |   4 +-
 e2e/rust/e2e-openshift.sh                     | 184 ---------
 e2e/with-kube-gateway.sh                      | 380 ++++++++++++------
 skills/debug-openshell-cluster/SKILL.md       |   3 +
 tasks/test.toml                               |   3 -
 8 files changed, 431 insertions(+), 320 deletions(-)
 create mode 100644 deploy/helm/openshell/ci/values-openshift-e2e.yaml
 create mode 100644 deploy/helm/openshell/ci/values-openshift-scc.yaml
 delete mode 100755 e2e/rust/e2e-openshift.sh

diff --git a/TESTING.md b/TESTING.md
index 199fa35416..93213ea24a 100644
--- a/TESTING.md
+++ b/TESTING.md
@@ -232,6 +232,116 @@ Secrets and Vault storage backends one at a time:
 mise run e2e:kubernetes:credential-drivers
 ```
 
+### Kubernetes E2E (`e2e/rust/e2e-kubernetes.sh`)
+
+Kubernetes e2e tests deploy an OpenShell gateway into a real Kubernetes cluster
+via Helm, port-forward the gateway, and run the Rust e2e suite against it.
+
+Run with an ephemeral k3d cluster (macOS; created and torn down automatically):
+
+```shell
+mise run e2e:kubernetes
+```
+
+Target an existing cluster (kind, k3d, or OpenShift):
+
+```shell
+OPENSHELL_E2E_KUBE_CONTEXT=my-context mise run e2e:kubernetes
+```
+
+Scope to a single test for local debugging:
+
+```shell
+OPENSHELL_E2E_KUBE_TEST=smoke mise run e2e:kubernetes
+```
+
+**OpenShift**: when the target cluster exposes the `route.openshift.io` API
+group, the harness automatically applies SCC-compatible Helm overrides and
+grants the required SCCs. No extra flags or steps are needed.
+
+On a **remote** cluster, drop the `e2e-host-gateway` feature. Those tests rely
+on the sandbox-side `host.openshell.internal` alias reaching the machine running
+the tests, which is unreachable from pods on a remote cluster, so they fail.
+Left enabled, the `host_gateway_alias` suite fails because
+`host.openshell.internal` does not resolve inside the pod, so the gateway
+SSRF-denies the request (`DNS resolution failed` / `ssrf_denied`) — a networking
+property of remote pods, not a gateway or transport fault. Override
+`OPENSHELL_E2E_KUBERNETES_FEATURES` to exclude it:
+
+```shell
+OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
+  OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
+  mise run e2e:kubernetes
+```
+
+On an existing cluster the harness builds the CLI from your branch but pulls the
+**published** gateway/supervisor image (default tag `latest`). The CLI and the
+image can therefore be different versions. If tests fail because of this version
+difference — for example, sandbox tests fail with `Pod exists with phase: Failed`
+or connect-based tests stall because the deployed image predates a feature your
+branch CLI needs — set `IMAGE_TAG` to an image that matches your branch.
+
+The `latest` tag lags to the last semver release, so it is often older than
+`main`. Two better choices:
+
+- `IMAGE_TAG=dev` — a floating tag that tracks the latest `main` build. Good for
+  an ad-hoc run when your branch is close to `main` HEAD. Because it floats, two
+  runs on different days can pull different images, so it is not reproducible.
+- **Pin the exact commit your branch is based on** — deterministic and immune to
+  a floating tag moving. Published tags are the full 40-char git SHA (semver tags
+  without a `v` prefix also exist but only for released versions):
+
+```shell
+OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
+  OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
+  IMAGE_TAG=$(git rev-parse "$(git merge-base HEAD upstream/main)") \
+  mise run e2e:kubernetes
+```
+
+To pin a specific released version, use its semver tag without a `v` prefix
+(`0.0.115`, not `v0.0.115`):
+
+```shell
+OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
+  OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
+  IMAGE_TAG=0.0.115 \
+  mise run e2e:kubernetes
+```
+
+A semver tag matches a released commit, which may be behind `main`; if your
+branch CLI needs a newer feature, pin the SHA of your branch's base instead.
+
+Confirm a tag exists before relying on it:
+`skopeo inspect docker://ghcr.io/nvidia/openshell/gateway:<tag>`.
+
+`IMAGE_TAG` sets only the gateway/supervisor image; the CLI under test is always
+built from your branch. To validate against images from your exact commit
+instead, build and push them and point `OPENSHELL_REGISTRY`/`IMAGE_TAG` at them.
+
+Available task variants:
+
+| Task | Purpose |
+|---|---|
+| `e2e:kubernetes` | Default Rust e2e against Helm-deployed gateway |
+| `e2e:kubernetes:db` | All database backend scenarios (SQLite + external PostgreSQL) |
+| `e2e:kubernetes:sidecar` | Supervisor sidecar topology overlay |
+| `e2e:kubernetes:credential-drivers` | Kubernetes Secrets and Vault credential storage |
+| `e2e:kubernetes:workspace-managed` | Managed workspace mode (auto-created namespaces) |
+| `e2e:kubernetes:workspace-operator` | Operator workspace mode (pre-provisioned namespaces) |
+| `e2e:kubernetes:v1alpha1` | Agent Sandbox v1alpha1 compatibility |
+| `e2e:kubernetes:external-driver` | External Kubernetes driver sidecar |
+
+Kubernetes e2e environment variables:
+
+| Variable | Purpose |
+|---|---|
+| `OPENSHELL_E2E_KUBE_CONTEXT` | kubectl context for an existing cluster (skips k3d creation) |
+| `OPENSHELL_E2E_KUBE_TEST` | Scope to a single test (e.g. `smoke`) |
+| `OPENSHELL_E2E_KUBE_EXTRA_VALUES` | Colon-separated additional Helm values files |
+| `OPENSHELL_E2E_KUBERNETES_FEATURES` | Cargo feature flags (default: `e2e,e2e-host-gateway,e2e-kubernetes`) |
+| `IMAGE_TAG` | Gateway/supervisor image tag (default: `latest` for existing clusters) |
+| `OPENSHELL_REGISTRY` | Image registry prefix (default: `ghcr.io/nvidia/openshell`) |
+
 Run a single test directly with cargo:
 
 ```shell
@@ -263,3 +373,5 @@ The harness (`e2e/rust/src/harness/`) provides:
 | `OPENSHELL_GATEWAY_ENDPOINT` | Run E2E tests against an existing plaintext HTTP gateway endpoint |
 | `OPENSHELL_E2E_DRIVER` | Driver name exported by the e2e gateway wrapper (`docker`, `podman`, or `vm`) |
 | `OPENSHELL_E2E_CREDENTIAL_DRIVERS` | Enables the Kubernetes credential-driver fixture path in `e2e/with-kube-gateway.sh` |
+| `OPENSHELL_E2E_KUBE_CONTEXT` | kubectl context for Kubernetes e2e (skips ephemeral k3d) |
+| `OPENSHELL_E2E_KUBE_TEST` | Scope Kubernetes e2e to a single test by name |
diff --git a/deploy/helm/openshell/ci/values-openshift-e2e.yaml b/deploy/helm/openshell/ci/values-openshift-e2e.yaml
new file mode 100644
index 0000000000..40d8ce9586
--- /dev/null
+++ b/deploy/helm/openshell/ci/values-openshift-e2e.yaml
@@ -0,0 +1,45 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# OpenShift overlay for the Kubernetes e2e harness.
+#
+# Bundles the OpenShift-specific settings the harness needs: the Route/mTLS
+# transport, and an image pull policy that avoids stale cached images.
+#
+# Route/mTLS transport: on OpenShift, `kubectl port-forward` stalls the SSH-relay
+# `sandbox connect` path (round-trip-heavy SSH over SPDY), so the harness drives
+# the gateway through a passthrough OpenShift Route with mTLS instead. This
+# overlay turns TLS back on (values-skaffold.yaml disables it), enables the Route,
+# and promotes the cert-verified caller to a dev principal.
+#
+# Image pull policy: force `Always` so runs against the `latest` upstream image
+# actually use it, instead of a stale copy cached on the cluster nodes.
+#
+# Layered by e2e/with-kube-gateway.sh AFTER ci/values-skaffold.yaml and
+# ci/values-openshift-scc.yaml when an OpenShift cluster is detected. The harness
+# supplies `openshiftRoute.host` and `pkiInitJob.serverDnsNames[0]` via --set at
+# install time (both are the cluster-derived Route hostname).
+#
+# Security: this is NOT an open gateway. `server.tls.clientCaSecretName` defaults
+# to `openshell-server-client-ca` and there is no OIDC, so `require_client_auth`
+# is true and mTLS is MANDATORY at the TLS handshake — a caller with only the
+# Route URL and no client certificate is rejected before any RPC. The passthrough
+# Route terminates TLS at the gateway pod, so this holds end-to-end.
+# `allowUnauthenticatedUsers` only promotes the already cert-verified caller to a
+# dev principal at the app layer (mtls_auth is unsupported with the Kubernetes
+# driver). Both are required together; the client certificate is the access gate.
+image:
+  pullPolicy: Always
+
+supervisor:
+  image:
+    pullPolicy: Always
+
+server:
+  disableTls: false
+  auth:
+    allowUnauthenticatedUsers: true
+
+openshiftRoute:
+  enabled: true
+  # host is supplied via --set at install time (cluster-derived Route hostname).
diff --git a/deploy/helm/openshell/ci/values-openshift-scc.yaml b/deploy/helm/openshell/ci/values-openshift-scc.yaml
new file mode 100644
index 0000000000..b7f37be6e0
--- /dev/null
+++ b/deploy/helm/openshell/ci/values-openshift-scc.yaml
@@ -0,0 +1,20 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# OpenShift SCC compatibility overlay. Removes the hardcoded runAsUser and
+# fsGroup so that OpenShift's restricted-v2 SCC can inject the namespace-
+# assigned UID/GID range. Layer after values.yaml:
+#   helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml
+#
+# The e2e Kubernetes harness applies this automatically when it detects an
+# OpenShift cluster (route.openshift.io API present).
+
+podSecurityContext: null
+
+securityContext:
+  runAsNonRoot: true
+  runAsUser: null
+  allowPrivilegeEscalation: false
+  capabilities:
+    drop:
+      - ALL
diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl
index 54d9e1ff99..25662518e8 100644
--- a/deploy/helm/openshell/templates/_gateway-workload.tpl
+++ b/deploy/helm/openshell/templates/_gateway-workload.tpl
@@ -34,8 +34,10 @@ spec:
         - host.docker.internal
         - host.openshell.internal
   {{- end }}
+  {{- with .Values.podSecurityContext }}
   securityContext:
-    {{- toYaml .Values.podSecurityContext | nindent 4 }}
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   containers:
     - name: openshell-gateway
       securityContext:
diff --git a/e2e/rust/e2e-openshift.sh b/e2e/rust/e2e-openshift.sh
deleted file mode 100755
index 639e6323a4..0000000000
--- a/e2e/rust/e2e-openshift.sh
+++ /dev/null
@@ -1,184 +0,0 @@
-#!/usr/bin/env bash
-# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-#
-# Validates all OpenShift database-backend scenarios against a live cluster.
-#
-# Prerequisites:
-#   - oc CLI authenticated to an OpenShift cluster
-#   - helm 3.x installed
-#
-# Usage:
-#   mise run e2e:openshift
-#   e2e/rust/e2e-openshift.sh [--chart-path ./deploy/helm/openshell] [--image-tag dev]
-
-set -euo pipefail
-
-ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-# shellcheck source=e2e/support/gateway-common.sh
-source "${ROOT}/e2e/support/gateway-common.sh"
-
-CHART_PATH="${CHART_PATH:-./deploy/helm/openshell}"
-NAMESPACE="openshell"
-RELEASE="openshell"
-IMAGE_TAG="${IMAGE_TAG:-dev}"
-WAIT_TIMEOUT="120s"
-PASSED=0
-FAILED=0
-SCENARIOS=()
-EXTERNAL_PG_SECRET="my-pg-credentials"
-EXTERNAL_PG_SERVICE="openshell-e2e-postgres"
-EXTERNAL_PG_SERVICE_ACCOUNT="openshell-e2e-postgres"
-EXTERNAL_PG_PASSWORD="openshell-e2e-postgres"
-EXTERNAL_PG_DATABASE="openshell"
-EXTERNAL_PG_USERNAME="openshell"
-EXTERNAL_PG_MANIFEST="${ROOT}/e2e/kubernetes/postgres-fixture.yaml"
-
-while [[ $# -gt 0 ]]; do
-  case $1 in
-    --chart-path) CHART_PATH="$2"; shift 2 ;;
-    --image-tag)  IMAGE_TAG="$2"; shift 2 ;;
-    --namespace)  NAMESPACE="$2"; shift 2 ;;
-    *) echo "Unknown option: $1" >&2; exit 1 ;;
-  esac
-done
-
-# --- helpers ----------------------------------------------------------------
-
-log()  { echo "==> $*"; }
-pass() { log "PASS: $1"; PASSED=$((PASSED + 1)); SCENARIOS+=("PASS  $1"); }
-fail() { log "FAIL: $1 — $2"; FAILED=$((FAILED + 1)); SCENARIOS+=("FAIL  $1: $2"); }
-
-wait_for_ready() {
-  local label="$1" timeout="$2"
-  if oc wait pod -n "$NAMESPACE" -l "$label" --for=condition=Ready --timeout="$timeout" 2>/dev/null; then
-    return 0
-  fi
-  return 1
-}
-
-cleanup_release() {
-  log "Cleaning up release $RELEASE"
-  helm uninstall "$RELEASE" -n "$NAMESPACE" --wait 2>/dev/null || true
-  # Wait for pods to terminate
-  for i in $(seq 1 30); do
-    if [ -z "$(oc get pods -n "$NAMESPACE" -l "app.kubernetes.io/instance=$RELEASE" --no-headers 2>/dev/null)" ]; then
-      break
-    fi
-    sleep 2
-  done
-  # Clean up PVCs left by StatefulSets
-  oc delete pvc -n "$NAMESPACE" -l "app.kubernetes.io/instance=$RELEASE" --wait=false 2>/dev/null || true
-}
-
-deploy_external_pg() {
-  local pg_uri
-
-  log "Deploying standalone PostgreSQL as external database..."
-  oc create serviceaccount "$EXTERNAL_PG_SERVICE_ACCOUNT" -n "$NAMESPACE" 2>/dev/null || true
-  oc adm policy add-scc-to-user anyuid -z "$EXTERNAL_PG_SERVICE_ACCOUNT" -n "$NAMESPACE" >/dev/null
-
-  oc apply -n "$NAMESPACE" -f "$EXTERNAL_PG_MANIFEST"
-  oc rollout status "deployment/${EXTERNAL_PG_SERVICE}" -n "$NAMESPACE" --timeout="$WAIT_TIMEOUT"
-
-  pg_uri="postgresql://${EXTERNAL_PG_USERNAME}:${EXTERNAL_PG_PASSWORD}@${EXTERNAL_PG_SERVICE}.${NAMESPACE}.svc.cluster.local:5432/${EXTERNAL_PG_DATABASE}"
-  log "Creating existing Secret with PostgreSQL credentials..."
-  oc delete secret "$EXTERNAL_PG_SECRET" -n "$NAMESPACE" --ignore-not-found >/dev/null 2>&1 || true
-  oc create secret generic "$EXTERNAL_PG_SECRET" -n "$NAMESPACE" \
-    --from-literal=uri="$pg_uri"
-}
-
-cleanup_external_pg() {
-  oc delete -n "$NAMESPACE" -f "$EXTERNAL_PG_MANIFEST" --ignore-not-found 2>/dev/null || true
-  oc delete secret "$EXTERNAL_PG_SECRET" -n "$NAMESPACE" --ignore-not-found 2>/dev/null || true
-  oc adm policy remove-scc-from-user anyuid -z "$EXTERNAL_PG_SERVICE_ACCOUNT" \
-    -n "$NAMESPACE" 2>/dev/null || true
-}
-
-verify_gateway() {
-  local scenario="$1"
-  if wait_for_ready "app.kubernetes.io/name=openshell,app.kubernetes.io/instance=$RELEASE" "$WAIT_TIMEOUT"; then
-    # Check the pod is actually running (not CrashLoopBackOff)
-    local phase
-    phase=$(oc get pod -n "$NAMESPACE" -l "app.kubernetes.io/name=openshell,app.kubernetes.io/instance=$RELEASE" \
-      -o jsonpath='{.items[0].status.phase}' 2>/dev/null)
-    if [ "$phase" = "Running" ]; then
-      pass "$scenario"
-    else
-      fail "$scenario" "pod phase is $phase, expected Running"
-    fi
-  else
-    local status
-    status=$(oc get pods -n "$NAMESPACE" -l "app.kubernetes.io/name=openshell" --no-headers 2>/dev/null || echo "no pods found")
-    fail "$scenario" "gateway pod not ready within $WAIT_TIMEOUT ($status)"
-  fi
-}
-
-# --- setup ------------------------------------------------------------------
-
-log "Setting up namespace $NAMESPACE"
-oc create ns "$NAMESPACE" 2>/dev/null || true
-oc adm policy add-scc-to-user privileged -z "${RELEASE}-sandbox" -n "$NAMESPACE"
-
-OPENSHIFT_FLAGS=(
-  --set server.disableTls=true
-  --set "server.telemetryEnabled=${OPENSHELL_TELEMETRY_ENABLED}"
-  --set podSecurityContext.fsGroup=null
-  --set securityContext.runAsUser=null
-  --set image.tag="$IMAGE_TAG"
-)
-
-# --- scenario 1: SQLite (default, no postgres) -----------------------------
-
-SCENARIO="SQLite (default)"
-log "Testing: $SCENARIO"
-cleanup_release
-
-helm install "$RELEASE" "$CHART_PATH" -n "$NAMESPACE" \
-  "${OPENSHIFT_FLAGS[@]}"
-
-verify_gateway "$SCENARIO"
-cleanup_release
-
-# --- scenario 2: External PostgreSQL with existing Secret -------------------
-
-SCENARIO="External PostgreSQL (externalDbSecret)"
-log "Testing: $SCENARIO"
-cleanup_release
-
-deploy_external_pg
-
-# Install OpenShell pointing at the existing Secret
-helm install "$RELEASE" "$CHART_PATH" -n "$NAMESPACE" \
-  "${OPENSHIFT_FLAGS[@]}" \
-  --set server.externalDbSecret="$EXTERNAL_PG_SECRET"
-
-verify_gateway "$SCENARIO"
-
-# Cleanup external postgres and secret
-cleanup_release
-cleanup_external_pg
-
-# --- teardown ---------------------------------------------------------------
-
-log "Removing SCC binding and namespace"
-oc adm policy remove-scc-from-user privileged -z "${RELEASE}-sandbox" -n "$NAMESPACE" 2>/dev/null || true
-cleanup_external_pg
-oc delete ns "$NAMESPACE" --wait=false 2>/dev/null || true
-
-# --- summary ----------------------------------------------------------------
-
-echo ""
-echo "========================================"
-echo "  Test Summary"
-echo "========================================"
-for s in "${SCENARIOS[@]}"; do
-  echo "  $s"
-done
-echo "----------------------------------------"
-echo "  Passed: $PASSED  Failed: $FAILED"
-echo "========================================"
-
-if [ "$FAILED" -gt 0 ]; then
-  exit 1
-fi
diff --git a/e2e/with-kube-gateway.sh b/e2e/with-kube-gateway.sh
index bd7f246d24..2aa0ef7304 100755
--- a/e2e/with-kube-gateway.sh
+++ b/e2e/with-kube-gateway.sh
@@ -13,8 +13,20 @@
 #       Create a local k3d cluster via tasks/scripts/helm-k3s-local.sh, install
 #       the chart, port-forward, and tear the cluster down on exit.
 #
-# Helm e2e currently uses plaintext gateway traffic (ci/values-skaffold.yaml).
-# The certgen hook still runs so the gateway has sandbox JWT signing keys.
+# On vanilla Kubernetes, Helm e2e talks to the gateway in plaintext over
+# `kubectl port-forward` (ci/values-skaffold.yaml). The certgen hook still runs
+# so the gateway has sandbox JWT signing keys.
+#
+# On OpenShift, that port-forward is too slow for `sandbox connect`. That command
+# opens an SSH session to the gateway, and SSH needs many small back-and-forth
+# messages to set up. Each one has to travel through the port-forward tunnel, so
+# the connection never finishes and the test times out. To avoid this, on
+# OpenShift the harness reaches the gateway through a normal network path: a
+# passthrough OpenShift Route, secured with mandatory mTLS
+# (ci/values-openshift-e2e.yaml).
+#
+# Every OpenShift-specific branch below is gated on OPENSHIFT_DETECTED, so the
+# vanilla-Kubernetes path stays exactly the same.
 #
 # Set OPENSHELL_E2E_KUBE_EXTRA_VALUES to one or more colon-separated Helm values
 # files, relative to the repository root or absolute, to layer additional chart
@@ -94,6 +106,12 @@ VAULT_CHART_VERSION="${OPENSHELL_E2E_OPENBAO_CHART_VERSION:-0.28.3}"
 VAULT_DEV_ROOT_TOKEN="${OPENSHELL_E2E_VAULT_DEV_ROOT_TOKEN:-root}"
 CORPORATE_PROXY_FIXTURE_DEPLOYED=0
 CORPORATE_PROXY_FIXTURE_SECRET="openshell-e2e-proxy-auth"
+OPENSHIFT_DETECTED=0
+OPENSHIFT_SANDBOX_SCC_GRANTED=0
+OPENSHIFT_ROUTE_HOST=""
+# Temp dir holding the client mTLS material extracted from openshell-client-tls
+# for the OpenShift Route transport. Removed by cleanup().
+OPENSHIFT_PKI_DIR="${WORKDIR}/openshift-pki"
 
 # Isolate CLI/SDK gateway metadata from the developer's real config.
 export XDG_CONFIG_HOME="${WORKDIR}/config"
@@ -134,6 +152,13 @@ deploy_postgres_fixture() {
     kctl create namespace "${NAMESPACE}"
   fi
 
+  if [ "${OPENSHIFT_DETECTED}" = "1" ]; then
+    echo "Granting anyuid SCC to ${EXTERNAL_PG_FIXTURE_SERVICE} for OpenShift..."
+    oc adm policy add-scc-to-user anyuid \
+      --context "${KUBE_CONTEXT}" \
+      -z "${EXTERNAL_PG_FIXTURE_SERVICE}" -n "${NAMESPACE}"
+  fi
+
   kctl -n "${NAMESPACE}" apply -f "${EXTERNAL_PG_FIXTURE_MANIFEST}"
   EXTERNAL_PG_FIXTURE_DEPLOYED=1
   EXTERNAL_PG_FIXTURE_SECRET="${secret_name}"
@@ -158,6 +183,13 @@ cleanup_postgres_fixture() {
   kctl -n "${NAMESPACE}" delete secret "${secret_name}" \
     --ignore-not-found >/dev/null 2>&1 || true
 
+  if [ "${OPENSHIFT_DETECTED}" = "1" ]; then
+    oc adm policy remove-scc-from-user anyuid \
+      --context "${KUBE_CONTEXT}" \
+      -z "${EXTERNAL_PG_FIXTURE_SERVICE}" -n "${NAMESPACE}" \
+      2>/dev/null || true
+  fi
+
   EXTERNAL_PG_FIXTURE_DEPLOYED=0
   EXTERNAL_PG_FIXTURE_SECRET=""
 }
@@ -255,6 +287,20 @@ cleanup() {
       --ignore-not-found >/dev/null 2>&1 || true
   fi
 
+  if [ "${OPENSHIFT_SANDBOX_SCC_GRANTED}" = "1" ]; then
+    oc adm policy remove-scc-from-user privileged \
+      --context "${KUBE_CONTEXT}" \
+      -z openshell-sandbox -n "${NAMESPACE}" \
+      2>/dev/null || true
+    OPENSHIFT_SANDBOX_SCC_GRANTED=0
+  fi
+
+  # Remove the extracted client mTLS material (also covered by the WORKDIR sweep
+  # below, but drop the private key promptly and explicitly).
+  if [ -n "${OPENSHIFT_PKI_DIR}" ]; then
+    rm -rf "${OPENSHIFT_PKI_DIR}" 2>/dev/null || true
+  fi
+
   # Sweep managed-mode and operator-mode workspace namespaces before
   # uninstalling the Helm release (ClusterRole still needed for deletion).
   if command -v kubectl >/dev/null 2>&1 && [ -n "${KUBE_CONTEXT}" ]; then
@@ -327,6 +373,15 @@ scenario_cleanup_release() {
     -l "app.kubernetes.io/instance=${RELEASE_NAME}" --wait=false 2>/dev/null || true
 }
 
+scenario_record_failure() {
+  local scenario_label="$1"
+  local reason="$2"
+  DB_FAILED=$((DB_FAILED + 1))
+  DB_SCENARIOS_SUMMARY+=("FAIL  ${scenario_label}: ${reason}")
+  scenario_stop_portforward
+  scenario_cleanup_release
+}
+
 scenario_deploy_external_pg() {
   echo "==> Deploying standalone PostgreSQL as external database..."
   deploy_postgres_fixture my-pg-credentials
@@ -363,84 +418,33 @@ run_scenario() {
     --wait --timeout 5m
   HELM_INSTALLED=1
 
-  LOCAL_PORT="$(e2e_pick_port)"
-  echo "Starting kubectl port-forward svc/openshell ${LOCAL_PORT}:8080..."
-  kctl -n "${NAMESPACE}" port-forward "svc/openshell" \
-    "${LOCAL_PORT}:8080" >"${PORTFORWARD_LOG}" 2>&1 &
-  PORTFORWARD_PID=$!
-
-  local elapsed=0 pf_timeout=30
-  while [ "${elapsed}" -lt "${pf_timeout}" ]; do
-    if ! kill -0 "${PORTFORWARD_PID}" 2>/dev/null; then
-      echo "ERROR: kubectl port-forward exited before becoming reachable" >&2
-      cat "${PORTFORWARD_LOG}" >&2 || true
-      DB_FAILED=$((DB_FAILED + 1))
-      DB_SCENARIOS_SUMMARY+=("FAIL  ${scenario_label}: port-forward died")
-      scenario_stop_portforward
-      scenario_cleanup_release
+  if [ "${OPENSHIFT_DETECTED}" = "1" ]; then
+    # OpenShift: reach the gateway over the passthrough Route with mTLS instead
+    # of port-forward (which stalls the SSH-relay connect suites).
+    if ! openshift_register_route_gateway; then
+      scenario_record_failure "${scenario_label}" "Route/mTLS setup failed"
       return
     fi
-    if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${LOCAL_PORT}"; then
-      break
+  else
+    # Vanilla Kubernetes: reach the gateway in plaintext over port-forward.
+    if ! start_grpc_portforward; then
+      scenario_record_failure "${scenario_label}" "port-forward failed"
+      return
     fi
-    sleep 1
-    elapsed=$((elapsed + 1))
-  done
-  if [ "${elapsed}" -ge "${pf_timeout}" ]; then
-    echo "ERROR: port-forward did not accept TCP within ${pf_timeout}s" >&2
-    cat "${PORTFORWARD_LOG}" >&2 || true
-    DB_FAILED=$((DB_FAILED + 1))
-    DB_SCENARIOS_SUMMARY+=("FAIL  ${scenario_label}: port-forward timeout")
-    scenario_stop_portforward
-    scenario_cleanup_release
-    return
+    GATEWAY_NAME="openshell-e2e-kube-${LOCAL_PORT}"
+    GATEWAY_ENDPOINT="http://127.0.0.1:${LOCAL_PORT}"
+    e2e_register_plaintext_gateway \
+      "${XDG_CONFIG_HOME}" \
+      "${GATEWAY_NAME}" \
+      "${GATEWAY_ENDPOINT}" \
+      "${LOCAL_PORT}"
   fi
 
-  HEALTH_LOCAL_PORT="$(e2e_pick_port)"
-  local workload_ref
-  workload_ref="$(kube_workload_ref "${RELEASE_NAME}")"
-  echo "Starting kubectl port-forward ${workload_ref} ${HEALTH_LOCAL_PORT}:health..."
-  kctl -n "${NAMESPACE}" port-forward "${workload_ref}" \
-    "${HEALTH_LOCAL_PORT}:health" >"${PORTFORWARD_HEALTH_LOG}" 2>&1 &
-  PORTFORWARD_HEALTH_PID=$!
-
-  elapsed=0
-  while [ "${elapsed}" -lt "${pf_timeout}" ]; do
-    if ! kill -0 "${PORTFORWARD_HEALTH_PID}" 2>/dev/null; then
-      echo "ERROR: kubectl health port-forward exited before becoming reachable" >&2
-      cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true
-      DB_FAILED=$((DB_FAILED + 1))
-      DB_SCENARIOS_SUMMARY+=("FAIL  ${scenario_label}: health port-forward died")
-      scenario_stop_portforward
-      scenario_cleanup_release
-      return
-    fi
-    if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${HEALTH_LOCAL_PORT}/healthz"; then
-      break
-    fi
-    sleep 1
-    elapsed=$((elapsed + 1))
-  done
-  if [ "${elapsed}" -ge "${pf_timeout}" ]; then
-    echo "ERROR: health port-forward did not accept TCP within ${pf_timeout}s" >&2
-    cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true
-    DB_FAILED=$((DB_FAILED + 1))
-    DB_SCENARIOS_SUMMARY+=("FAIL  ${scenario_label}: health port-forward timeout")
-    scenario_stop_portforward
-    scenario_cleanup_release
+  if ! start_health_portforward; then
+    scenario_record_failure "${scenario_label}" "health port-forward failed"
     return
   fi
 
-  export OPENSHELL_E2E_HEALTH_PORT="${HEALTH_LOCAL_PORT}"
-
-  GATEWAY_NAME="openshell-e2e-kube-${LOCAL_PORT}"
-  GATEWAY_ENDPOINT="http://127.0.0.1:${LOCAL_PORT}"
-  e2e_register_plaintext_gateway \
-    "${XDG_CONFIG_HOME}" \
-    "${GATEWAY_NAME}" \
-    "${GATEWAY_ENDPOINT}" \
-    "${LOCAL_PORT}"
-
   export OPENSHELL_GATEWAY="${GATEWAY_NAME}"
   export OPENSHELL_E2E_DRIVER="kubernetes"
   # Kubernetes e2e runs against k3d/kind-style Docker-backed clusters. Host
@@ -493,6 +497,128 @@ configure_fixture_container_engine() {
   export CONTAINER_ENGINE="${selected_engine}"
 }
 
+# OpenShift only: extract the client mTLS material, wait for the passthrough
+# Route to serve mTLS, assert that a certless caller is rejected at the TLS
+# handshake, and register an mTLS CLI gateway pointing at the Route.
+#
+# Sets GATEWAY_NAME and GATEWAY_ENDPOINT on success. Returns non-zero on failure
+# (unreachable Route or a certless request that was NOT rejected — a security
+# hole). Reads OPENSHIFT_ROUTE_HOST and OPENSHIFT_PKI_DIR.
+openshift_register_route_gateway() {
+  local pki_dir="${OPENSHIFT_PKI_DIR}"
+
+  rm -rf "${pki_dir}"
+  mkdir -p "${pki_dir}/client"
+
+  echo "Extracting client mTLS material from secret openshell-client-tls..."
+  kctl -n "${NAMESPACE}" get secret openshell-client-tls \
+    -o jsonpath='{.data.ca\.crt}' | base64 -d >"${pki_dir}/ca.crt"
+  kctl -n "${NAMESPACE}" get secret openshell-client-tls \
+    -o jsonpath='{.data.tls\.crt}' | base64 -d >"${pki_dir}/client/tls.crt"
+  kctl -n "${NAMESPACE}" get secret openshell-client-tls \
+    -o jsonpath='{.data.tls\.key}' | base64 -d >"${pki_dir}/client/tls.key"
+
+  # Wait until an mTLS request to the Route completes the TLS handshake. Helm
+  # --wait already made the gateway pod Ready; this only covers the short window
+  # while the OpenShift router loads the new Route.
+  echo "Waiting for Route https://${OPENSHIFT_ROUTE_HOST} to serve mTLS..."
+  local elapsed=0 timeout=180
+  while [ "${elapsed}" -lt "${timeout}" ]; do
+    if curl -s --max-time 10 -o /dev/null \
+         --cacert "${pki_dir}/ca.crt" \
+         --cert "${pki_dir}/client/tls.crt" \
+         --key "${pki_dir}/client/tls.key" \
+         "https://${OPENSHIFT_ROUTE_HOST}/"; then
+      break
+    fi
+    sleep 3
+    elapsed=$((elapsed + 3))
+  done
+  if [ "${elapsed}" -ge "${timeout}" ]; then
+    echo "ERROR: Route ${OPENSHIFT_ROUTE_HOST} did not serve mTLS within ${timeout}s" >&2
+    return 1
+  fi
+
+  # Security gate: a caller with no client certificate MUST be rejected during
+  # the TLS handshake (clientCaSecretName set + no OIDC => mTLS mandatory).
+  if curl -sk --max-time 10 -o /dev/null "https://${OPENSHIFT_ROUTE_HOST}/"; then
+    echo "ERROR: SECURITY HOLE — gateway accepted a certless request over the Route" >&2
+    return 1
+  fi
+  echo "OK: Route reachable over mTLS; certless request rejected at TLS."
+
+  GATEWAY_NAME="openshell-e2e-openshift"
+  GATEWAY_ENDPOINT="https://${OPENSHIFT_ROUTE_HOST}"
+  e2e_register_mtls_gateway \
+    "${XDG_CONFIG_HOME}" \
+    "${GATEWAY_NAME}" \
+    "${GATEWAY_ENDPOINT}" \
+    "$(e2e_endpoint_port "${GATEWAY_ENDPOINT}")" \
+    "${pki_dir}"
+}
+
+# Start `kubectl port-forward svc/openshell` for the gRPC endpoint and wait for
+# it to accept TCP. Sets LOCAL_PORT and PORTFORWARD_PID. Prints the port-forward
+# log and returns non-zero on failure. Used for the vanilla-Kubernetes transport
+# (the OpenShift transport uses openshift_register_route_gateway instead).
+start_grpc_portforward() {
+  LOCAL_PORT="$(e2e_pick_port)"
+  echo "Starting kubectl port-forward svc/openshell ${LOCAL_PORT}:8080..."
+  kctl -n "${NAMESPACE}" port-forward "svc/openshell" \
+    "${LOCAL_PORT}:8080" >"${PORTFORWARD_LOG}" 2>&1 &
+  PORTFORWARD_PID=$!
+
+  local elapsed=0 timeout=30
+  while [ "${elapsed}" -lt "${timeout}" ]; do
+    if ! kill -0 "${PORTFORWARD_PID}" 2>/dev/null; then
+      echo "ERROR: kubectl port-forward exited before becoming reachable" >&2
+      cat "${PORTFORWARD_LOG}" >&2 || true
+      return 1
+    fi
+    if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${LOCAL_PORT}"; then
+      return 0
+    fi
+    sleep 1
+    elapsed=$((elapsed + 1))
+  done
+  echo "ERROR: port-forward did not accept TCP within ${timeout}s" >&2
+  cat "${PORTFORWARD_LOG}" >&2 || true
+  return 1
+}
+
+# Start `kubectl port-forward` for the health endpoint and wait for /healthz.
+# Sets HEALTH_LOCAL_PORT and PORTFORWARD_HEALTH_PID and exports
+# OPENSHELL_E2E_HEALTH_PORT. Used on both cluster types: the OpenShift Route
+# targets grpc only, and the health endpoint is not the SSH path so port-forward
+# is fine for it. Prints the log and returns non-zero on failure.
+start_health_portforward() {
+  HEALTH_LOCAL_PORT="$(e2e_pick_port)"
+  local workload_ref
+  workload_ref="$(kube_workload_ref "${RELEASE_NAME}")"
+  echo "Starting kubectl port-forward ${workload_ref} ${HEALTH_LOCAL_PORT}:health..."
+  kctl -n "${NAMESPACE}" port-forward "${workload_ref}" \
+    "${HEALTH_LOCAL_PORT}:health" >"${PORTFORWARD_HEALTH_LOG}" 2>&1 &
+  PORTFORWARD_HEALTH_PID=$!
+
+  local elapsed=0 timeout=30
+  while [ "${elapsed}" -lt "${timeout}" ]; do
+    if ! kill -0 "${PORTFORWARD_HEALTH_PID}" 2>/dev/null; then
+      echo "ERROR: kubectl health port-forward exited before becoming reachable" >&2
+      cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true
+      return 1
+    fi
+    if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${HEALTH_LOCAL_PORT}/healthz"; then
+      export OPENSHELL_E2E_HEALTH_PORT="${HEALTH_LOCAL_PORT}"
+      return 0
+    fi
+    sleep 1
+    elapsed=$((elapsed + 1))
+  done
+  echo "ERROR: health port-forward did not accept TCP within ${timeout}s" >&2
+  cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true
+  return 1
+}
+
 require_cmd helm
 require_cmd kubectl
 require_cmd curl
@@ -738,6 +864,41 @@ if [ -n "${HOST_GATEWAY_IP}" ]; then
 fi
 
 helm_values_args=(--values "${ROOT}/deploy/helm/openshell/ci/values-skaffold.yaml")
+if kctl api-resources --api-group=route.openshift.io --no-headers 2>/dev/null | grep -q .; then
+  OPENSHIFT_DETECTED=1
+  echo "OpenShift detected — applying SCC-compatible security context overrides."
+  helm_values_args+=(--values "${ROOT}/deploy/helm/openshell/ci/values-openshift-scc.yaml")
+
+  if ! command -v oc >/dev/null 2>&1; then
+    echo "ERROR: oc CLI is required for OpenShift SCC management but was not found." >&2
+    exit 2
+  fi
+
+  kctl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kctl apply -f -
+
+  echo "Granting privileged SCC to openshell-sandbox in namespace ${NAMESPACE}..."
+  oc adm policy add-scc-to-user privileged \
+    --context "${KUBE_CONTEXT}" \
+    -z openshell-sandbox -n "${NAMESPACE}"
+  OPENSHIFT_SANDBOX_SCC_GRANTED=1
+
+  # Drive the gateway through a passthrough Route with mTLS instead of
+  # port-forward. The Route host is deterministic: OpenShift serves any name
+  # under the cluster ingress (apps) domain via the router's wildcard, so we
+  # bake "<release>-<namespace>.<apps-domain>" into the server cert SANs before
+  # the Route exists.
+  APPS_DOMAIN="$(kctl get ingresses.config/cluster -o jsonpath='{.spec.domain}')"
+  if [ -z "${APPS_DOMAIN}" ]; then
+    echo "ERROR: could not resolve the OpenShift cluster ingress domain." >&2
+    exit 2
+  fi
+  OPENSHIFT_ROUTE_HOST="${RELEASE_NAME}-${NAMESPACE}.${APPS_DOMAIN}"
+  echo "Using OpenShift Route host ${OPENSHIFT_ROUTE_HOST}."
+
+  helm_values_args+=(--values "${ROOT}/deploy/helm/openshell/ci/values-openshift-e2e.yaml")
+  helm_extra_args+=(--set "openshiftRoute.host=${OPENSHIFT_ROUTE_HOST}")
+  helm_extra_args+=(--set "pkiInitJob.serverDnsNames[0]=${OPENSHIFT_ROUTE_HOST}")
+fi
 if [ "${OPENSHELL_E2E_KUBE_CORPORATE_PROXY:-0}" = "1" ]; then
   if [ -z "${HOST_GATEWAY_IP}" ]; then
     echo "ERROR: corporate proxy e2e requires a host gateway IP for host.openshell.internal" >&2
@@ -866,68 +1027,23 @@ else
       --docker-password=e2e-password
   fi
 
-  LOCAL_PORT="$(e2e_pick_port)"
-  echo "Starting kubectl port-forward svc/openshell ${LOCAL_PORT}:8080..."
-  kctl -n "${NAMESPACE}" port-forward "svc/openshell" \
-    "${LOCAL_PORT}:8080" >"${PORTFORWARD_LOG}" 2>&1 &
-  PORTFORWARD_PID=$!
-
-  elapsed=0
-  timeout=30
-  while [ "${elapsed}" -lt "${timeout}" ]; do
-    if ! kill -0 "${PORTFORWARD_PID}" 2>/dev/null; then
-      echo "ERROR: kubectl port-forward exited before becoming reachable" >&2
-      cat "${PORTFORWARD_LOG}" >&2 || true
-      exit 1
-    fi
-    if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${LOCAL_PORT}"; then
-      break
-    fi
-    sleep 1
-    elapsed=$((elapsed + 1))
-  done
-  if [ "${elapsed}" -ge "${timeout}" ]; then
-    echo "ERROR: port-forward did not accept TCP within ${timeout}s" >&2
-    cat "${PORTFORWARD_LOG}" >&2 || true
-    exit 1
-  fi
-
-  HEALTH_LOCAL_PORT="$(e2e_pick_port)"
-  WORKLOAD_REF="$(kube_workload_ref "${RELEASE_NAME}")"
-  echo "Starting kubectl port-forward ${WORKLOAD_REF} ${HEALTH_LOCAL_PORT}:health..."
-  kctl -n "${NAMESPACE}" port-forward "${WORKLOAD_REF}" \
-    "${HEALTH_LOCAL_PORT}:health" >"${PORTFORWARD_HEALTH_LOG}" 2>&1 &
-  PORTFORWARD_HEALTH_PID=$!
-
-  elapsed=0
-  timeout=30
-  while [ "${elapsed}" -lt "${timeout}" ]; do
-    if ! kill -0 "${PORTFORWARD_HEALTH_PID}" 2>/dev/null; then
-      echo "ERROR: kubectl health port-forward exited before becoming reachable" >&2
-      cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true
-      exit 1
-    fi
-    if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${HEALTH_LOCAL_PORT}/healthz"; then
-      break
-    fi
-    sleep 1
-    elapsed=$((elapsed + 1))
-  done
-  if [ "${elapsed}" -ge "${timeout}" ]; then
-    echo "ERROR: health port-forward did not accept TCP within ${timeout}s" >&2
-    cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true
-    exit 1
+  if [ "${OPENSHIFT_DETECTED}" = "1" ]; then
+    # OpenShift: reach the gateway over the passthrough Route with mTLS so the
+    # SSH-relay `sandbox connect` suites work (port-forward stalls them).
+    openshift_register_route_gateway || exit 1
+  else
+    # Vanilla Kubernetes: reach the gateway in plaintext over port-forward.
+    start_grpc_portforward || exit 1
+    GATEWAY_NAME="openshell-e2e-kube-${LOCAL_PORT}"
+    GATEWAY_ENDPOINT="http://127.0.0.1:${LOCAL_PORT}"
+    e2e_register_plaintext_gateway \
+      "${XDG_CONFIG_HOME}" \
+      "${GATEWAY_NAME}" \
+      "${GATEWAY_ENDPOINT}" \
+      "${LOCAL_PORT}"
   fi
 
-  export OPENSHELL_E2E_HEALTH_PORT="${HEALTH_LOCAL_PORT}"
-
-  GATEWAY_NAME="openshell-e2e-kube-${LOCAL_PORT}"
-  GATEWAY_ENDPOINT="http://127.0.0.1:${LOCAL_PORT}"
-  e2e_register_plaintext_gateway \
-    "${XDG_CONFIG_HOME}" \
-    "${GATEWAY_NAME}" \
-    "${GATEWAY_ENDPOINT}" \
-    "${LOCAL_PORT}"
+  start_health_portforward || exit 1
 
   export OPENSHELL_GATEWAY="${GATEWAY_NAME}"
   export OPENSHELL_E2E_DRIVER="kubernetes"
diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md
index 8e86bc0643..7d47b18065 100644
--- a/skills/debug-openshell-cluster/SKILL.md
+++ b/skills/debug-openshell-cluster/SKILL.md
@@ -74,6 +74,7 @@ Use gateway metadata, deployment values, or the user's setup notes to identify t
 | Docker | Gateway process logs, Docker daemon health, sandbox containers, image pulls. |
 | Podman | Podman socket, rootless networking, sandbox containers, image pulls. |
 | Kubernetes | Helm release, gateway workload, service, secrets, sandbox pods, events. |
+| OpenShift | Same as Kubernetes, plus SecurityContextConstraints (SCCs) and, for external access, an OpenShift `Route`. Detect OpenShift by the presence of the `route.openshift.io` API group (`oc api-resources --api-group=route.openshift.io`). |
 | VM | VM driver logs, rootfs availability, host virtualization support. |
 | Extension | External driver process, Unix socket ownership/mode, configured driver name, capability handshake, gateway logs. |
 
@@ -686,6 +687,8 @@ configuration — check that the gateway spawned the driver binary you expect
 | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>` |
 | Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` |
 | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` |
+| OpenShift gateway pod fails to start with an SCC/`runAsUser` error (e.g. `unable to validate against any security context constraint`) | Chart's default `podSecurityContext`/`securityContext` hardcodes `runAsUser`/`fsGroup`, which the restricted-v2 SCC rejects; it must instead inject the namespace-assigned UID/GID range | `oc -n openshell describe pod <pod>`; deploy with `podSecurityContext: null` and clear `securityContext.runAsUser` (see `deploy/helm/openshell/ci/values-openshift-scc.yaml`) |
+| OpenShift sandbox pod fails to start (`unable to validate against any security context constraint`) | The `openshell-sandbox` service account lacks the privileged SCC it needs | `oc adm policy add-scc-to-user privileged -z openshell-sandbox -n openshell`; remove with `remove-scc-from-user` when done |
 | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways/<name>/mtls/` |
 | Edge or OIDC gateway returns `Unauthenticated` | Stored login expired, audience/scopes mismatch, or gateway auth configuration changed | `openshell gateway info`, `openshell gateway login <name>`, gateway auth logs |
 | Gateway fails before serving health after enabling an interceptor | Interceptor endpoint unavailable or manifest/binding validation failed | Gateway and interceptor logs; interceptor socket; `binding_policy`, phases, and failure policy |
diff --git a/tasks/test.toml b/tasks/test.toml
index 4a5cda0890..bb1fa2e9ab 100644
--- a/tasks/test.toml
+++ b/tasks/test.toml
@@ -268,6 +268,3 @@ env = { OPENSHELL_E2E_DOCKER_GPU = "1", OPENSHELL_E2E_DOCKER_TEST = "gpu", OPENS
 depends = ["e2e:conformance:build"]
 run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-docker.sh"
 
-["e2e:openshift"]
-description = "Run OpenShift database-backend integration scenarios against a live cluster (requires oc CLI authenticated to an OpenShift cluster)"
-run = "e2e/rust/e2e-openshift.sh"

From eeb0ad282fe0ed706fac3c94d04922303f9dbb08 Mon Sep 17 00:00:00 2001
From: Jorge Garcia Oncins <jgarciao@redhat.com>
Date: Mon, 31 Aug 2026 17:44:40 +0200
Subject: [PATCH 2/2] fix(e2e): harden OpenShift SCC cleanup, mTLS gate, and
 Route timeout

Track the anyuid SCC grant for the PostgreSQL fixture with a dedicated
OPENSHIFT_POSTGRES_SCC_GRANTED flag set before the fixture apply, so a
failed apply no longer leaks the binding; cleanup now revokes it whenever
the grant succeeded, independent of deploy state.

Validate the Route server cert in the certless security gate (curl
--cacert instead of -k) and classify curl's exit code so only a TLS
client-auth rejection (35/56) counts as the expected certless rejection;
an unrelated DNS/timeout/TLS failure now fails loudly instead of masking
a potential mTLS hole.

Raise the OpenShift Route timeout in the e2e overlay. The default HAProxy
Route timeout is 30s, which severed long-lived transfers (large sandbox
upload/download, SSH-relay `sandbox connect`) mid-stream and failed the
sync e2e tests. Set both haproxy.router.openshift.io/timeout and
timeout-tunnel to 300s: a passthrough Route proxies in TCP mode, so
timeout-tunnel governs the established tunnel while timeout covers the
pre-tunnel phase.

Document the OpenShift transport exception, oc prerequisites and SCC
grants, and make the skopeo tag-check example copy-safe in TESTING.md.

Signed-off-by: Jorge Garcia Oncins <jgarciao@redhat.com>
---
 TESTING.md                                    | 22 +++++++---
 .../openshell/ci/values-openshift-e2e.yaml    |  8 ++++
 e2e/with-kube-gateway.sh                      | 40 +++++++++++++++----
 3 files changed, 58 insertions(+), 12 deletions(-)

diff --git a/TESTING.md b/TESTING.md
index 93213ea24a..14ec3c2055 100644
--- a/TESTING.md
+++ b/TESTING.md
@@ -235,7 +235,9 @@ mise run e2e:kubernetes:credential-drivers
 ### Kubernetes E2E (`e2e/rust/e2e-kubernetes.sh`)
 
 Kubernetes e2e tests deploy an OpenShell gateway into a real Kubernetes cluster
-via Helm, port-forward the gateway, and run the Rust e2e suite against it.
+via Helm and run the Rust e2e suite against it. On vanilla Kubernetes the harness
+reaches the gateway through `kubectl port-forward`; on OpenShift it instead uses a
+passthrough Route secured with mandatory mTLS (see the OpenShift note below).
 
 Run with an ephemeral k3d cluster (macOS; created and torn down automatically):
 
@@ -256,8 +258,14 @@ OPENSHELL_E2E_KUBE_TEST=smoke mise run e2e:kubernetes
 ```
 
 **OpenShift**: when the target cluster exposes the `route.openshift.io` API
-group, the harness automatically applies SCC-compatible Helm overrides and
-grants the required SCCs. No extra flags or steps are needed.
+group, the harness automatically applies SCC-compatible Helm overrides, grants
+the required SCCs (`privileged` to `openshell-sandbox`, and `anyuid` to the
+PostgreSQL fixture for DB scenarios), and drives the gateway through a
+passthrough Route with mandatory mTLS instead of port-forward. No extra flags are
+needed, but `oc` must be installed and authenticated against the target cluster
+with permission to modify SCC bindings (`oc adm policy add-scc-to-user`) — the
+harness exits early if `oc` is missing. The SCC grants and extracted client
+mTLS material are removed during cleanup, including on failure or interrupt.
 
 On a **remote** cluster, drop the `e2e-host-gateway` feature. Those tests rely
 on the sandbox-side `host.openshell.internal` alias reaching the machine running
@@ -311,8 +319,12 @@ OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
 A semver tag matches a released commit, which may be behind `main`; if your
 branch CLI needs a newer feature, pin the SHA of your branch's base instead.
 
-Confirm a tag exists before relying on it:
-`skopeo inspect docker://ghcr.io/nvidia/openshell/gateway:<tag>`.
+Confirm a tag exists before relying on it (set `TAG` to the tag you plan to use):
+
+```shell
+TAG=0.0.115
+skopeo inspect "docker://ghcr.io/nvidia/openshell/gateway:${TAG}"
+```
 
 `IMAGE_TAG` sets only the gateway/supervisor image; the CLI under test is always
 built from your branch. To validate against images from your exact commit
diff --git a/deploy/helm/openshell/ci/values-openshift-e2e.yaml b/deploy/helm/openshell/ci/values-openshift-e2e.yaml
index 40d8ce9586..d2aaf0fac8 100644
--- a/deploy/helm/openshell/ci/values-openshift-e2e.yaml
+++ b/deploy/helm/openshell/ci/values-openshift-e2e.yaml
@@ -43,3 +43,11 @@ server:
 openshiftRoute:
   enabled: true
   # host is supplied via --set at install time (cluster-derived Route hostname).
+  # The default HAProxy Route timeout is 30s, which severs long-lived transfers
+  # (large sandbox upload/download, SSH-relay `sandbox connect`) mid-stream. Raise
+  # both the connection timeout and the passthrough tunnel timeout so these paths
+  # survive. Passthrough Routes proxy in TCP mode, so timeout-tunnel governs the
+  # established tunnel while timeout covers the pre-tunnel phase.
+  annotations:
+    haproxy.router.openshift.io/timeout: 300s
+    haproxy.router.openshift.io/timeout-tunnel: 300s
diff --git a/e2e/with-kube-gateway.sh b/e2e/with-kube-gateway.sh
index 2aa0ef7304..8a9eaf3df2 100755
--- a/e2e/with-kube-gateway.sh
+++ b/e2e/with-kube-gateway.sh
@@ -108,6 +108,7 @@ CORPORATE_PROXY_FIXTURE_DEPLOYED=0
 CORPORATE_PROXY_FIXTURE_SECRET="openshell-e2e-proxy-auth"
 OPENSHIFT_DETECTED=0
 OPENSHIFT_SANDBOX_SCC_GRANTED=0
+OPENSHIFT_POSTGRES_SCC_GRANTED=0
 OPENSHIFT_ROUTE_HOST=""
 # Temp dir holding the client mTLS material extracted from openshell-client-tls
 # for the OpenShift Route transport. Removed by cleanup().
@@ -157,6 +158,9 @@ deploy_postgres_fixture() {
     oc adm policy add-scc-to-user anyuid \
       --context "${KUBE_CONTEXT}" \
       -z "${EXTERNAL_PG_FIXTURE_SERVICE}" -n "${NAMESPACE}"
+    # Record the grant before applying the fixture so cleanup revokes it even if
+    # the apply below fails and EXTERNAL_PG_FIXTURE_DEPLOYED is never set.
+    OPENSHIFT_POSTGRES_SCC_GRANTED=1
   fi
 
   kctl -n "${NAMESPACE}" apply -f "${EXTERNAL_PG_FIXTURE_MANIFEST}"
@@ -183,11 +187,12 @@ cleanup_postgres_fixture() {
   kctl -n "${NAMESPACE}" delete secret "${secret_name}" \
     --ignore-not-found >/dev/null 2>&1 || true
 
-  if [ "${OPENSHIFT_DETECTED}" = "1" ]; then
+  if [ "${OPENSHIFT_POSTGRES_SCC_GRANTED}" = "1" ]; then
     oc adm policy remove-scc-from-user anyuid \
       --context "${KUBE_CONTEXT}" \
       -z "${EXTERNAL_PG_FIXTURE_SERVICE}" -n "${NAMESPACE}" \
       2>/dev/null || true
+    OPENSHIFT_POSTGRES_SCC_GRANTED=0
   fi
 
   EXTERNAL_PG_FIXTURE_DEPLOYED=0
@@ -274,7 +279,8 @@ cleanup() {
     fi
   fi
 
-  if [ "${EXTERNAL_PG_FIXTURE_DEPLOYED}" = "1" ]; then
+  if [ "${EXTERNAL_PG_FIXTURE_DEPLOYED}" = "1" ] \
+     || [ "${OPENSHIFT_POSTGRES_SCC_GRANTED}" = "1" ]; then
     cleanup_postgres_fixture "${EXTERNAL_PG_FIXTURE_SECRET}"
   fi
 
@@ -541,11 +547,31 @@ openshift_register_route_gateway() {
 
   # Security gate: a caller with no client certificate MUST be rejected during
   # the TLS handshake (clientCaSecretName set + no OIDC => mTLS mandatory).
-  if curl -sk --max-time 10 -o /dev/null "https://${OPENSHIFT_ROUTE_HOST}/"; then
-    echo "ERROR: SECURITY HOLE — gateway accepted a certless request over the Route" >&2
-    return 1
-  fi
-  echo "OK: Route reachable over mTLS; certless request rejected at TLS."
+  #
+  # Validate the server cert with --cacert (no -k) and inspect curl's exit code
+  # so an unrelated TLS/DNS/timeout failure is not silently accepted as "certless
+  # rejected". Only a handshake abort by the server (no client cert presented)
+  # counts as the expected rejection.
+  local certless_rc=0
+  curl -s --max-time 10 -o /dev/null \
+    --cacert "${pki_dir}/ca.crt" \
+    "https://${OPENSHIFT_ROUTE_HOST}/" || certless_rc=$?
+  case "${certless_rc}" in
+    0)
+      echo "ERROR: SECURITY HOLE — gateway accepted a certless request over the Route" >&2
+      return 1
+      ;;
+    35 | 56)
+      # 35 CURLE_SSL_CONNECT_ERROR / 56 CURLE_RECV_ERROR: the server aborted the
+      # TLS handshake because no client certificate was presented — the expected
+      # mTLS rejection.
+      echo "OK: Route reachable over mTLS; certless request rejected at TLS (curl ${certless_rc})."
+      ;;
+    *)
+      echo "ERROR: certless probe to ${OPENSHIFT_ROUTE_HOST} failed with curl exit ${certless_rc}, not a TLS client-auth rejection; cannot confirm mTLS is enforced" >&2
+      return 1
+      ;;
+  esac
 
   GATEWAY_NAME="openshell-e2e-openshift"
   GATEWAY_ENDPOINT="https://${OPENSHIFT_ROUTE_HOST}"