feat(otel-collector): per-signal table TTL + reconcile on existing tables

[ZeynelKoca]

What

Adds per-signal ClickHouse table TTLs and, opt-in, reconciles TTL on already-existing tables. Implements #1311.

Why

Today HYPERDX_OTEL_EXPORTER_TABLES_TTL (#1720) sets one TTL for every signal, and it only applies at table creation (CREATE TABLE IF NOT EXISTS). So:

• retention can't differ per signal — e.g. keeping logs/traces for 6 months (compliance) while metrics stay at 30 days, and
• changing it has no effect on an already-provisioned deployment (existing tables keep their old TTL).

Why this matters for compliance. Frameworks like SOC 2 (and ISO 27001, HIPAA, PCI-DSS) commonly require security-relevant logs, and often traces, to be retained for 6–12 months — while high-volume metrics are kept short to control storage cost. That is inherently a per-signal retention policy, and it must apply to the tables a running deployment already has, not just fresh installs. #1311 asked for exactly this (per-signal 90 days / 6 months / 1 year / 5 years). Without this change, operators on any serious/regulated deployment have to hand-run ALTER TABLE ... MODIFY TTL on every table after each deploy or upgrade — easy to get wrong and easy to silently drift out of compliance.

Changes

• Per-signal env vars, each falling back to HYPERDX_OTEL_EXPORTER_TABLES_TTL so existing setups are unchanged:

  • HYPERDX_OTEL_EXPORTER_LOGS_TTL
  • HYPERDX_OTEL_EXPORTER_TRACES_TTL
  • HYPERDX_OTEL_EXPORTER_METRICS_TTL
  • HYPERDX_OTEL_EXPORTER_SESSIONS_TTL

  The seed SQL now uses per-signal ${*_TTL} macros (logs also covers its rollup/compat tables; traces likewise; metrics its tables; sessions its own).

• HYPERDX_OTEL_EXPORTER_RECONCILE_TABLE_TTL (default false). When enabled, after seeding the migrate tool reconciles TTL on existing managed tables: it reads each table's current TTL from system.tables and, only when the retention differs, runs ALTER TABLE ... MODIFY TTL <anchor + new interval> SETTINGS materialize_ttl_after_modify = 0. It is diff-guarded via a syntax-agnostic duration comparison (so toIntervalDay(30) vs INTERVAL 30 DAY doesn't cause churn), preserves each table's timestamp anchor, and is non-fatal (a failed ALTER logs a warning and never blocks startup).

Backward compatibility

• No new env set → identical to today (all signals use TABLES_TTL; create-only).
• Reconcile is off by default, so an upgrade never silently rewrites an existing deployment's TTLs.

Testing

• go test ./cmd/migrate/ — added unit tests for per-signal config resolution/fallback, seed macro substitution, table→signal classification, TTL-clause extraction (incl. column-level TTL present), interval parsing (both toIntervalX(N) and INTERVAL N UNIT forms, day/hour/minute/second), and the reconcile decision planTTLReconcile — a pure function covering the cross-syntax diff-guard, extend vs shrink, and multi-interval refusal.
• go vet and gofmt clean.

Empirically verified

Ran the migrate binary (built from this branch) against a throwaway clickhouse-server:25.7-alpine container:

STEP 1 — seed with TABLES_TTL=30d                → all tables 30d

STEP 2 — LOGS_TTL=180d TRACES_TTL=180d RECONCILE_TABLE_TTL=true   (extend)
[seed] reconcile: otel_logs -> TTL toDateTime(Timestamp) + toIntervalDay(180)
[seed] reconcile: otel_logs_kv_rollup_15m -> TTL Timestamp + toIntervalDay(180)
[seed] reconcile: otel_traces -> TTL toDate(Timestamp) + toIntervalDay(180)
[seed] reconcile: otel_traces_kv_rollup_15m -> TTL Timestamp + toIntervalDay(180)
[seed] reconcile: examined=10 matched=10 altered=4 skipped=6 warned=0 failed=0

STEP 3 — METRICS_TTL=7d RECONCILE_TABLE_TTL=true                  (shrink)
[seed] reconcile: otel_metrics_sum -> TTL toDateTime(TimeUnix) + toIntervalDay(7)
[seed] ... (5 metrics tables)
[seed] reconcile: examined=10 matched=10 altered=5 skipped=5 warned=0 failed=0

STEP 4 — identical re-run                                         (idempotent)
(no output — a no-op reconcile logs nothing)

Direction-aware SETTINGS, read back from system.query_log:
  otel_logs           MODIFY TTL ... toIntervalDay(180)   materialize_ttl_after_modify = 1   (extend: keeps existing data)
  otel_traces         MODIFY TTL ... toIntervalDay(180)   materialize_ttl_after_modify = 1
  otel_metrics_sum    MODIFY TTL ... toIntervalDay(7)     materialize_ttl_after_modify = 0   (shrink: no bulk delete on startup)

Confirms: per-signal TTLs apply; existing logs/traces tables (incl. rollups) are reconciled with timestamp anchors preserved; metrics/sessions are isolated; extending materializes (existing data retained for the new period) while shrinking stays lazy (no startup delete); and an identical re-run alters nothing.

Implements #1311.

[changeset-bot]

🦋 Changeset detected

Latest commit: 166644f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/otel-collector	Minor
@hyperdx/api	Minor
@hyperdx/app	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@ZeynelKoca is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

This PR adds per-signal retention settings for ClickHouse telemetry tables. The main changes are:

• Separate TTL settings for logs, traces, metrics, and sessions.
• Backward-compatible fallback to the existing table TTL setting.
• Optional reconciliation of TTLs on existing managed tables.
• Safe skipping of compound TTL policies and unchanged retention values.
• Updated seed SQL and unit tests for the new behavior.

Confidence Score: 5/5

The latest changes look safe to merge.

• The compound TTL path now skips policies with multiple recognized intervals instead of modifying one rule.
• The updated tests cover the new reconciliation decision and the main TTL syntax variants.
• No separate blocking issue remains within this follow-up review scope.

Important Files Changed

Filename	Overview
packages/otel-collector/cmd/migrate/main.go	Adds per-signal TTL configuration, TTL parsing and planning, and optional reconciliation for existing ClickHouse tables.
packages/otel-collector/cmd/migrate/main_test.go	Adds coverage for configuration fallback, macro replacement, table classification, TTL parsing, and reconciliation decisions.
docker/otel-collector/schema/seed/00003_otel_metrics.sql	Updates metric tables to use the metrics-specific TTL setting.
docker/otel-collector/schema/seed/00004_hyperdx_sessions.sql	Updates the sessions table to use the sessions-specific TTL setting.

_{Reviews (5): Last reviewed commit: "feat(otel-collector): per-signal table T..." | Re-trigger Greptile}