Operational alerting is implemented end to end but has no receiver, so every alert currently terminates in the process that raised it.
Current state (verified in prod, 2026-08-09)
SCALE_ALERT_WEBHOOK_URL — not set. fly secrets list holds only DATABASE_URL, CLOAK_KEY, SECRET_KEY_BASE, ECTO_IPV6, OPENAI_API_KEY, ANTHROPIC_API_KEY, TENANT_AUDIT_KEY_ECOMMERCE_FRIENDLY.SCALE_ALERTS_ENABLED = "false" in fly.toml — deliberate, and the comment there says why: OFF means the ScaleAlerts child is not started at all, so no threshold is evaluated, logged or POSTed. Prometheus on METRICS_PORT remains the degradation signal.
This is a coherent, documented state, not a bug. The gap is that it has no follow-up attached, and the consequence is easy to misread.
Why it needs closing
alerted: true on an ingestion anomaly records that a ScaleAlertDeliveryWorker job was ENQUEUED, not that anyone was told. With no URL the worker logs scale_alert_webhook_url is unset at delivery time and skips. An operator reading the anomaly list sees five rows marked alerted and reasonably concludes someone was notified.
Concretely today: five capture_silence anomalies have carried alerted: true since mid-July (23 to 31 days) with no human aware of them. They turned out to be completed one-off imports rather than broken feeds, so nothing was lost this time. The next one will look identical.
The always-on Logger.error in IngestionHealthWorker still fires, so the information exists in the Fly logs. Nobody reads Fly logs on a cadence that would catch a month-old silence.
What to do
- Decide the receiver (Slack incoming webhook, PagerDuty, or a generic endpoint).
- Set BOTH halves in the same change: the
SCALE_ALERT_WEBHOOK_URL secret AND SCALE_ALERTS_ENABLED = "true" in fly.toml. ScaleAlerts.config_status/2 flags either half-done direction and /health/ready goes red naming the setting to fix, so a one-sided change is caught rather than silently half-working.
- Confirm
/health/ready returns 200 after the change (it is the deploy smoke gate and is deliberately not wired into fly.toml's continuous check, so it will not depool a node either way).
Optional, separate
Consider whether alerted should mean DELIVERED rather than ENQUEUED, or whether the field should be split. Not required to close this issue, and not worth doing until a receiver exists.
Operational alerting is implemented end to end but has no receiver, so every alert currently terminates in the process that raised it.
Current state (verified in prod, 2026-08-09)
SCALE_ALERT_WEBHOOK_URL— not set.fly secrets listholds only DATABASE_URL, CLOAK_KEY, SECRET_KEY_BASE, ECTO_IPV6, OPENAI_API_KEY, ANTHROPIC_API_KEY, TENANT_AUDIT_KEY_ECOMMERCE_FRIENDLY.SCALE_ALERTS_ENABLED = "false"infly.toml— deliberate, and the comment there says why: OFF means theScaleAlertschild is not started at all, so no threshold is evaluated, logged or POSTed. Prometheus onMETRICS_PORTremains the degradation signal.This is a coherent, documented state, not a bug. The gap is that it has no follow-up attached, and the consequence is easy to misread.
Why it needs closing
alerted: trueon an ingestion anomaly records that aScaleAlertDeliveryWorkerjob was ENQUEUED, not that anyone was told. With no URL the worker logsscale_alert_webhook_url is unset at delivery timeand skips. An operator reading the anomaly list sees five rows marked alerted and reasonably concludes someone was notified.Concretely today: five
capture_silenceanomalies have carriedalerted: truesince mid-July (23 to 31 days) with no human aware of them. They turned out to be completed one-off imports rather than broken feeds, so nothing was lost this time. The next one will look identical.The always-on
Logger.errorinIngestionHealthWorkerstill fires, so the information exists in the Fly logs. Nobody reads Fly logs on a cadence that would catch a month-old silence.What to do
SCALE_ALERT_WEBHOOK_URLsecret ANDSCALE_ALERTS_ENABLED = "true"infly.toml.ScaleAlerts.config_status/2flags either half-done direction and/health/readygoes red naming the setting to fix, so a one-sided change is caught rather than silently half-working./health/readyreturns 200 after the change (it is the deploy smoke gate and is deliberately not wired intofly.toml's continuous check, so it will not depool a node either way).Optional, separate
Consider whether
alertedshould mean DELIVERED rather than ENQUEUED, or whether the field should be split. Not required to close this issue, and not worth doing until a receiver exists.