[HDX-4879] Restore inline custom OTEL collector config support

[wrn14897]

Summary

Restores the otel.customConfig capability that was lost in the v3 migration to the official OpenTelemetry Collector subchart (#188), as a new global.otelCollector.customConfig value.

Linear: HDX-4879

global:
  otelCollector:
    customConfig: |
      receivers:
        hostmetrics:
          collection_interval: 5s
          scrapers:
            cpu:
      service:
        pipelines:
          metrics/hostmetrics:
            receivers: [hostmetrics]
            processors: [memory_limiter, batch]
            exporters: [clickhouse]

How it works

The subchart's values.schema.json sets additionalProperties: false, so the value can't live under otel-collector:. global.* is free-form and visible to both parent templates and the subchart's tpl contexts, which enables a fully turnkey design:

1. New ConfigMap clickstack-otel-custom-config (templates/otel-collector/custom-config.yaml) rendered when the value is set and the collector is enabled
2. Pre-wired mount via otel-collector.extraVolumes/extraVolumeMounts in values.yaml — directory mount at /etc/otelcol-contrib/custom with optional: true, so it is a no-op when unset (directory mount avoids the kubelet subPath failure mode for missing optional ConfigMaps)
3. Conditional env var CUSTOM_OTELCOL_CONFIG_FILE added to the shared clickstack-config ConfigMap (the collector already consumes it via extraEnvsFrom; this is the only conditional env path since subchart extraEnvs is static). An explicit hyperdx.config.CUSTOM_OTELCOL_CONFIG_FILE takes precedence
4. Auto-restart on config change via a checksum/custom-config pod annotation (podAnnotations is tpl'd and map-merged in the subchart, so user annotations still merge)

The collector image honors CUSTOM_OTELCOL_CONFIG_FILE in both OpAMP supervisor mode (default) and standalone mode. Note the subchart's native config:/relay.yaml is ignored by the ClickStack image entrypoint, which is why this env-var path is required.

Known limitation

If users override otel-collector.extraVolumes/extraVolumeMounts, Helm replaces the pre-wired lists — documented in values.yaml, the example README, and UPGRADE.md.

Changes

• charts/clickstack/values.yaml — new global.otelCollector.customConfig + pre-wired volume/mount/checksum annotation
• charts/clickstack/templates/otel-collector/custom-config.yaml — new ConfigMap template
• charts/clickstack/templates/hyperdx/configmap.yaml — conditional CUSTOM_OTELCOL_CONFIG_FILE
• charts/clickstack/tests/otel-collector-custom-config_test.yaml — rewritten from vestigial generic assertions to 9 real tests, including subchart deployment wiring assertions
• docs/UPGRADE.md — migration path for the old otel.customConfig
• examples/otel-custom-config/ — worked example (validated in CI)
• Changeset (minor)

Test plan

• helm unittest charts/clickstack — 187 tests pass (9 new)
• Default helm template diff vs main is a behavioral no-op (only inert optional volume/mount + constant checksum annotation)
• Render with custom config: ConfigMap, env var, checksum, and mount all present; custom config round-trips as valid YAML
• otel-collector.enabled=false + customConfig set → no ConfigMap rendered
• All example values files render

[changeset-bot]

🦋 Changeset detected

Latest commit: b6854ee

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
helm-charts	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Deep Review

✅ No critical issues found. The default render (with customConfig unset) is a verified behavioral no-op — the and guards are falsy on the empty-string default, so no ConfigMap and no env var are emitted, and only an inert optional: true volume/mount plus a constant checksum annotation are added. The multi-line | nindent 4 block scalar round-trips as valid YAML, and the checksum/custom-config tpl mechanism is confirmed working by the passing default-checksum unit test.

🟡 P2 -- recommended

• charts/clickstack/templates/hyperdx/configmap.yaml:11 -- this PR adds a new dereference of .Values.global.otelCollector.customConfig into the always-rendered core clickstack-config ConfigMap, so an explicit global.otelCollector: null override aborts the entire chart render with nil pointer evaluating interface {}.customConfig (Go-template and evaluates all args eagerly, so no site short-circuits); the same unguarded dereference exists at custom-config.yaml:1/:11 and values.yaml:419.
  • Fix: Nil-guard the intermediate map at all four sites, e.g. (.Values.global.otelCollector | default dict).customConfig.
  • _{correctness, adversarial}
• charts/clickstack/tests/otel-collector-custom-config_test.yaml:117 -- the new checksum/custom-config podAnnotation is claimed to merge with user-provided otel-collector.podAnnotations, but no test sets a user annotation and asserts both it and checksum/custom-config survive.
  • Fix: Add a deployment test that sets otel-collector.podAnnotations.<userKey> and asserts the rendered annotations contain both keys.

🔵 P3 nitpicks (4)

• charts/clickstack/templates/hyperdx/configmap.yaml:11 -- the CUSTOM_OTELCOL_CONFIG_FILE env branch guards only on customConfig (not on otel-collector.enabled), so with the collector disabled and customConfig set the env var is injected into clickstack-config while the target ConfigMap is not rendered; currently benign since nothing consumes it, but inconsistent with custom-config.yaml:1.
  • Fix: Also gate the branch on (index .Values "otel-collector" "enabled") to match the ConfigMap template.
  • _{correctness, testing, adversarial}
• charts/clickstack/values.yaml:431 -- a user override of otel-collector.extraVolumeMounts replaces the pre-wired list, dropping the custom-config mount while CUSTOM_OTELCOL_CONFIG_FILE still points at the now-unmounted path, risking a collector startup failure; this is documented as a known limitation but has no guard or test.
  • Fix: Add a note-backed test or consider surfacing a clearer failure when the mount is absent but the env var is set.
• charts/clickstack/tests/otel-collector-custom-config_test.yaml:44 -- the multi-line customConfig round-trip is only substring-checked via matchRegex on the first two nesting levels, so a nindent mis-indentation of deeper keys would not be caught.
  • Fix: Extend the assertion to pin the deeper service.pipelines block indentation.
• charts/clickstack/tests/otel-collector-custom-config_test.yaml:16 -- the shipped examples/otel-custom-config/values.yaml is exercised only by CI helm template, not by any helm-unittest suite, so it can silently drift from the templates.
  • Fix: Add a suite that renders with values: [<example file>] and asserts the ConfigMap, env var, and checksum.

---

Reviewers (4): correctness, testing, project-standards, adversarial.

Testing gaps:

• No test covers otel-collector.enabled=false + customConfig set against hyperdx/configmap.yaml (the disabled-collector env-var asymmetry).
• No test covers global.otelCollector set to null, which would exercise the nil-pointer render path.
• Changeset presence could not be verified in this environment (.changeset/ could not be enumerated); confirm a minor changeset exists per the repo's changeset convention before merge.