Conversation
whi-tw
force-pushed
the
whi-tw/3269-trace-queue-worker
branch
from
September 8, 2026 09:24
a8d3f5b to
8b4d98b
Compare
We previously disabled tracing for the queue worker because the SolidQueue polling loops created a flood of low-value spans in X-Ray, drowning out the traces we actually care about: 806192d Don't enable OpenTelemetry for background jobs This re-enables tracing with a dedicated ADOT sidecar for the queue worker task, configured to filter out the polling noise at the collector level before it reaches X-Ray. Key changes: - Queue worker gets its own ADOT sidecar with a filter/drop_spans processor that drops SolidQueue poll spans (forms-runner-queue DB queries, ActiveRecord.transaction, SolidQueue::* internal spans) - OTEL_SERVICE_NAME overridden to `forms-runner-queue-worker` so it appears as a distinct node in the X-Ray service map - OTEL_TRACES_SAMPLER_ARG set to 1.0 — the queue worker has low traffic so full sampling is appropriate; noise control is handled by the collector filter rather than the SDK sampler - A sha256 hash of the collector config is embedded as a Docker label on the sidecar container, so changes to filter expressions automatically trigger a new task definition revision and ECS redeployment on the next terraform apply - The same dockerLabels hash pattern is applied to the shared ecs-service module so the main app also benefits The corresponding forms-runner change switches ActiveJob span naming to :job_class so traces are labelled with the job class name (e.g. `SendSubmissionJob process`) rather than the queue name (e.g. `submissions process`)
whi-tw
force-pushed
the
whi-tw/3269-trace-queue-worker
branch
from
September 8, 2026 09:26
8b4d98b to
b0d9a11
Compare
whi-tw
marked this pull request as ready for review
September 8, 2026 09:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this pull request solve?
Trello card: https://trello.com/c/KnaPaYVl/3269-tracing-for-cron-jobs-queue-worker
Things to consider when reviewing
Reminders
If you've made changes to the deployer role (files in
modules/deployer-access):make <environment> forms/account applyon the relevant environments (dev,stagingand/orprod)apply-forms-terraform-<environment>pipelines have run successfully