feat: add opt-in OCI runtime diagnostics

Containerfile

@@ -40,6 +40,7 @@ COPY . .
 FROM builder AS build
 
 ARG FACT_VERSION
+ARG FACT_BUILD_SHA=unknown
 ARG CARGO_ARGS=""
 RUN --mount=type=cache,target=/root/.cargo/registry \
     --mount=type=cache,target=/app/target \

deploy/acs-fact-lab/README.md

@@ -0,0 +1,120 @@
+# ACS-controlled FACT validation lab
+
+This is a development lab for comparing raw FACT events with RHACS file
+activity policy results. RHACS policies are the only source of monitored paths;
+do not edit the generated `stackrox/fact-config` ConfigMap.
+
+See [RESULTS-2026-09-01.md](RESULTS-2026-09-01.md) for the captured validation
+counts and conclusions.
+
+The deployment was validated on `rc-dev-cluster` on 2026-09-01 with:
+
+- StackRox source `d5255a30c33` and Roxie `v0.4.9`
+- development main image `quay.io/rcochran/main:4.12.x-529-gc619d5385e`
+- FACT source `96190289b3f4d0233ae62f231c4614fbcde0f2b3`
+- FACT image `quay.io/rcochran/scratch@sha256:4e6968b475595baf96425bbe04f45e005a398ff7babb762cdbc6a9c0d0f95655`
+- SigNoz chart `0.139.0` from the earlier `deploy/fact-signoz` lab
+
+The Roxie overlay selects FACT's third operating mode:
+
+1. Sensor output only: normal ACS configuration.
+2. Sensor plus OCI diagnostics: add `FACT_OCI_RUNTIME_SPEC_DEBUG=true`.
+3. Sensor plus OCI diagnostics plus OTLP: also add `FACT_OTEL_ENDPOINT`.
+
+OCI diagnostics and OTLP are additive. FACT remains connected to Sensor and the
+diagnostic feature does not change event selection or filtering.
+
+## Deploy
+
+Prerequisites are `oc`, `jq`, `curl`, the adjacent `stackrox/stackrox` checkout,
+an existing SigNoz deployment in `observability`, and registry credentials for
+the private development images. Do not commit the registry config or Roxie
+environment file.
+
+```sh
+export KUBECONFIG=/path/to/infractl-cluster-artifacts/kubeconfig
+export DOCKER_CONFIG_JSON=/path/to/registry/config.json
+export ROXIE_ENVRC=/tmp/roxie-fact-lab.envrc
+
+deploy/acs-fact-lab/setup-acs.sh
+deploy/acs-fact-lab/configure-lab.sh
+deploy/acs-fact-lab/run-experiments.sh
+```
+
+If SigNoz needs to be rebuilt, follow `deploy/fact-signoz/README.md` through the
+SigNoz installation and static collector configuration, but do not deploy the
+old standalone FACT operator or `Fact` custom resource.
+
+The setup script creates the `quay-rcochran` pull secret from the supplied
+registry config before Roxie deploys Central. The environment file contains the
+generated ACS password and CA path; keep it private.
+
+## What the experiment proves
+
+The controlled container sequence emits create, writable open, permission
+change, two renames, and unlink. The exact same sequence runs against container
+rootfs and an EmptyDir mounted at `/tmp`.
+
+| Case | Raw FACT identity | ACS result |
+|---|---|---|
+| container rootfs | empty `host_path`; OCI `no_mount_match` | deployment alert; empty `actualPath` |
+| EmptyDir | empty `host_path`; OCI matched bind mount from `kubernetes.io~empty-dir` | deployment alert unless excluded |
+| direct `oc debug node` host write | populated `host_path`; `openshift.debug=true`; container ID present | deployment alert attributed to the transient debug pod |
+| host `systemd-run` write | populated `host_path`; no container ID | node alert attributed to the node |
+
+The process policy (`Process Name=chmod`) and operation policy (`File
+Operation=open`) each select only their matching event while the raw OTLP stream
+retains the complete sequence. The exclusion policy suppresses the EmptyDir
+deployment only; it does not alter FACT collection or OTLP.
+
+Node versus deployment evaluation is currently determined by Sensor enrichment:
+a nonempty deployment ID is a deployment event. It is not determined by whether
+`host_path` is populated. This is why `oc debug node` is visible as a deployment
+event while a detached host systemd unit is a node event.
+
+The OCI mount match for a chrooted debug process is currently `no_mount_match`:
+the observed path is `/var/tmp/...` after chroot while the OCI mount destination
+is `/host`. The independent inode mapping still supplies the correct host path.
+
+## Inspect
+
+Open the SigNoz Logs Explorer and filter on `service.name = fact`. Useful fields
+are:
+
+```text
+fact.build.sha
+file.path
+file.host_path
+process.executable.path
+k8s.namespace.name
+k8s.pod.name
+openshift.debug
+container.oci.config.status
+container.oci.mount.status
+container.oci.mount.destination
+container.oci.mount.source
+```
+
+For terminal output:
+
+```sh
+SINCE_UTC=2026-09-01T16:00:00Z deploy/acs-fact-lab/query-otel.sh
+deploy/acs-fact-lab/query-alerts.sh
+oc -n stackrox get configmap fact-config -o jsonpath='{.data.fact\.yml}'
+```
+
+The ACS UI shows the same results under Violations. Central and Sensor logs are
+also available with `oc -n stackrox logs deployment/central` and
+`oc -n stackrox logs deployment/sensor`.
+
+## Tear down
+
+The default removes only the six lab policies, workloads, and isolated host
+marker. Add `--acs` to remove Roxie-deployed ACS. Add `--signoz` only when its
+persistent telemetry data should also be deleted.
+
+```sh
+deploy/acs-fact-lab/teardown.sh
+deploy/acs-fact-lab/teardown.sh --acs
+deploy/acs-fact-lab/teardown.sh --acs --signoz
+```

deploy/acs-fact-lab/RESULTS-2026-09-01.md

@@ -0,0 +1,52 @@
+# Validated results — 2026-09-01
+
+The repeatable matrix ran at `2026-09-01T17:13:49Z` on
+`rc-dev-cluster`. All six collector pods were ready. Every FACT process was
+simultaneously connected to Sensor and exporting OTLP to SigNoz.
+
+Sensor compiled the six RHACS policies to this FACT configuration:
+
+```yaml
+paths:
+  - /tmp/acs-fact-lab-marker
+  - /var/tmp/acs-fact-host-marker
+```
+
+No FACT ConfigMap was edited by the experiment.
+
+## Raw FACT stream
+
+The run produced 16 marker events in SigNoz:
+
+| Source | Count | Operations | Host path | OCI result |
+|---|---:|---|---|---|
+| container rootfs | 6 | create, open, chmod, rename, rename, unlink | empty | parsed, `no_mount_match` |
+| EmptyDir at `/tmp` | 6 | create, open, chmod, rename, rename, unlink | empty | parsed, `matched` |
+| direct `oc debug node` | 2 | open, chmod | `/var/tmp/acs-fact-host-marker` | parsed, `no_mount_match`, `openshift.debug=true` |
+| detached host systemd unit | 2 | open, chmod | `/var/tmp/acs-fact-host-marker` | no container or OCI attributes |
+
+The EmptyDir match identified destination `/tmp` and source
+`/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/scratch`. This is
+the positive evidence that distinguishes it from container rootfs even though
+both have an empty protobuf `host_path`.
+
+## ACS results
+
+| Policy | Alerts | Violations |
+|---|---:|---|
+| Container marker activity | 2 | six for rootfs and six for EmptyDir |
+| Marker changed by chmod | 2 | one chmod for each container deployment |
+| Marker writable open only | 2 | one writable open for each container deployment |
+| Exclude EmptyDir writer deployment | 1 | six for rootfs; none for EmptyDir |
+| Node host marker activity | 1 | open and chmod from the host systemd unit |
+| Containerized host marker activity | 1 | open and chmod attributed to the transient debug pod |
+
+For both rootfs and EmptyDir, ACS `actualPath` was empty. For both host-write
+cases it was `/var/tmp/acs-fact-host-marker`. The direct debug process retained
+pod, namespace, image, container, executable, and node attribution after the
+transient pod was removed.
+
+These results establish that RHACS criteria and exclusions reduce violations at
+policy evaluation time while raw OTLP remains complete. They also establish
+that ACS node/deployment routing follows Sensor's deployment enrichment, not
+the presence of `host_path`.

deploy/acs-fact-lab/configure-lab.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+# shellcheck disable=SC1091
+source "${SCRIPT_DIR}/lib.sh"
+require_command oc
+require_command jq
+require_command curl
+require_kubeconfig
+load_roxie_env
+
+oc apply -f "${SCRIPT_DIR}/workloads.yaml"
+oc -n "${LAB_NAMESPACE}" rollout status deployment/rootfs-writer --timeout=5m
+oc -n "${LAB_NAMESPACE}" rollout status deployment/emptydir-writer --timeout=5m
+
+node=$(oc get nodes -l node-role.kubernetes.io/worker -o jsonpath='{.items[0].metadata.name}')
+oc -n "${LAB_NAMESPACE}" create configmap fact-lab-state \
+  --from-literal=node="${node}" \
+  --dry-run=client -o yaml | oc apply -f -
+
+# Seed the exact host inode before the policy reaches FACT. This makes host
+# scanner behavior deterministic and limits all host writes to /var/tmp.
+oc -n "${LAB_NAMESPACE}" debug "node/${node}" -- \
+  chroot /host sh -c 'touch /var/tmp/acs-fact-host-marker && chmod 600 /var/tmp/acs-fact-host-marker'
+
+while IFS= read -r policy_name; do
+  while IFS= read -r policy_id; do
+    [[ -z "${policy_id}" ]] || rox_api DELETE "/v1/policies/${policy_id}" >/dev/null
+  done < <(policy_ids_by_name "${policy_name}")
+done < <(jq -r '.[].name' "${SCRIPT_DIR}/policies.json")
+
+policy_dir=$(mktemp -d)
+trap 'rm -rf "${policy_dir}"' EXIT
+policy_count=$(jq 'length' "${SCRIPT_DIR}/policies.json")
+for index in $(seq 0 $((policy_count - 1))); do
+  policy_file="${policy_dir}/policy-${index}.json"
+  jq ".[${index}]" "${SCRIPT_DIR}/policies.json" >"${policy_file}"
+  rox_api POST /v1/policies "${policy_file}" | jq -r '"created policy: \(.name) [\(.id)]"'
+done
+
+wait_for_fact_paths
+oc -n stackrox get configmap fact-config -o jsonpath='{.data.fact\.yml}'
+
+for deployment in rootfs-writer emptydir-writer; do
+  query=$(jq -rn --arg value "Deployment:${deployment}" '$value|@uri')
+  for _ in $(seq 1 60); do
+    count=$(rox_api GET "/v1/deployments?query=${query}" | jq --arg name "${deployment}" '[.deployments[]? | select(.name == $name)] | length')
+    [[ "${count}" -gt 0 ]] && break
+    sleep 2
+  done
+  [[ "${count}" -gt 0 ]] || {
+    echo "ACS did not inventory deployment ${deployment}" >&2
+    exit 1
+  }
+done
+
+echo "lab policies, workloads, host marker, and FACT paths are ready"

deploy/acs-fact-lab/lib.sh

@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+LAB_NAMESPACE=acs-file-activity-lab
+ROXIE_ENVRC=${ROXIE_ENVRC:-/tmp/roxie-fact-lab.envrc}
+
+require_command() {
+  command -v "$1" >/dev/null 2>&1 || {
+    echo "required command not found: $1" >&2
+    exit 1
+  }
+}
+
+require_kubeconfig() {
+  if [[ -z "${KUBECONFIG:-}" || ! -f "${KUBECONFIG}" ]]; then
+    echo "set KUBECONFIG to the downloaded cluster kubeconfig" >&2
+    exit 1
+  fi
+}
+
+load_roxie_env() {
+  if [[ ! -f "${ROXIE_ENVRC}" ]]; then
+    echo "Roxie environment file not found: ${ROXIE_ENVRC}" >&2
+    exit 1
+  fi
+  set -a
+  # shellcheck disable=SC1090
+  source "${ROXIE_ENVRC}"
+  set +a
+  : "${ROX_ENDPOINT:?ROX_ENDPOINT is missing from the Roxie environment}"
+  : "${ROX_USERNAME:?ROX_USERNAME is missing from the Roxie environment}"
+  : "${ROX_ADMIN_PASSWORD:?ROX_ADMIN_PASSWORD is missing from the Roxie environment}"
+  : "${ROX_CA_CERT_FILE:?ROX_CA_CERT_FILE is missing from the Roxie environment}"
+}
+
+rox_api() {
+  local method=$1
+  local api_path=$2
+  local data_file=${3:-}
+  local endpoint_host=${ROX_ENDPOINT%:*}
+  local args=(
+    --silent --show-error --fail-with-body --noproxy '*'
+    --resolve "central.stackrox:443:${endpoint_host}"
+    --user "${ROX_USERNAME}:${ROX_ADMIN_PASSWORD}"
+    --cacert "${ROX_CA_CERT_FILE}"
+    --request "${method}"
+  )
+  if [[ -n "${data_file}" ]]; then
+    args+=(--header 'Content-Type: application/json' --data-binary "@${data_file}")
+  fi
+  curl "${args[@]}" "https://central.stackrox${api_path}"
+}
+
+policy_ids_by_name() {
+  local policy_name=$1
+  local query
+  query=$(jq -rn --arg value "Policy:${policy_name}" '$value|@uri')
+  rox_api GET "/v1/policies?query=${query}" | jq -r --arg name "${policy_name}" '.policies[]? | select(.name == $name) | .id'
+}
+
+wait_for_fact_paths() {
+  local config
+  for _ in $(seq 1 90); do
+    config=$(oc -n stackrox get configmap fact-config -o jsonpath='{.data.fact\.yml}' 2>/dev/null || true)
+    if grep -q '/tmp/acs-fact-lab-marker' <<<"${config}" && grep -q '/var/tmp/acs-fact-host-marker' <<<"${config}"; then
+      return 0
+    fi
+    sleep 2
+  done
+  echo "timed out waiting for Sensor to compile the lab paths into fact-config" >&2
+  return 1
+}
+
+lab_node() {
+  local node
+  node=$(oc -n "${LAB_NAMESPACE}" get configmap fact-lab-state -o jsonpath='{.data.node}' 2>/dev/null || true)
+  if [[ -z "${node}" ]]; then
+    node=$(oc get nodes -l node-role.kubernetes.io/worker -o jsonpath='{.items[0].metadata.name}')
+  fi
+  printf '%s\n' "${node}"
+}