Skip to content

Add OpenTelemetry metrics for job callbacks - #1060

Draft
epompeii wants to merge 1 commit into
u/ep/callback/firefrom
u/ep/callback/metrics
Draft

epompeii wants to merge 1 commit into
u/ep/callback/firefrom
u/ep/callback/metrics

Conversation

@epompeii

@epompeii epompeii commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Callbacks get OpenTelemetry metrics.

  • callback.attempt counts each attempt by its class: a status class, a timeout, a connection error, or a block with its reason.
  • callback.finish.duration and callback.finish.attempts record each callback that settles, by its outcome and failure reason: its latency from the start of its delivery, and the attempts it made over its life. Only a settle that committed is recorded, and both have bucket boundaries sized for their values.
  • callback.pending is an observable gauge of the callbacks waiting to be delivered, read from the database at each export on a read connection only if one is free, so a busy pool skips a point instead of stalling the export.

Every attribute comes from a fixed enum, so no attribute carries a URL, a header, an address, or any other value from the request.

Needed for #1046

@epompeii
epompeii marked this pull request as ready for review September 26, 2026 01:02
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

🐰 Bencher Report

ProjectBencher
Branchu/ep/callback/metrics
Testbedintel-v1
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
microseconds (µs)
(Result Δ%)
Upper Boundary
microseconds (µs)
(Limit %)
Adapter::Json📈 view plot
🚷 view threshold
5.17 µs
(+4.95%)Baseline: 4.92 µs
6.00 µs
(86.20%)
Adapter::Magic (JSON)📈 view plot
🚷 view threshold
4.97 µs
(+4.34%)Baseline: 4.76 µs
5.68 µs
(87.42%)
Adapter::Magic (Rust)📈 view plot
🚷 view threshold
27.63 µs
(+3.58%)Baseline: 26.67 µs
30.32 µs
(91.11%)
Adapter::Rust📈 view plot
🚷 view threshold
4.68 µs
(+15.48%)Baseline: 4.06 µs
6.35 µs
(73.70%)
Adapter::RustBench📈 view plot
🚷 view threshold
4.66 µs
(+15.06%)Baseline: 4.05 µs
6.33 µs
(73.61%)
🐰 View full continuous benchmarking report in Bencher

@epompeii
epompeii added this pull request to stack #1054 September 26, 2026 02:44
@epompeii
epompeii marked this pull request as draft September 26, 2026 03:30
@epompeii
epompeii marked this pull request as ready for review September 26, 2026 04:19
@epompeii
epompeii marked this pull request as draft September 26, 2026 05:18
@epompeii
epompeii force-pushed the u/ep/callback/metrics branch from 8251882 to 9312c58 Compare September 26, 2026 14:13
@epompeii
epompeii marked this pull request as ready for review September 26, 2026 14:49
@epompeii epompeii removed the docker label Sep 26, 2026
@epompeii
epompeii marked this pull request as draft September 26, 2026 17:40
@epompeii
epompeii force-pushed the u/ep/callback/metrics branch 2 times, most recently from f30c36d to 436cb6f Compare September 26, 2026 20:21
@epompeii
epompeii force-pushed the u/ep/callback/metrics branch 2 times, most recently from ef1a727 to a1ed80b Compare September 27, 2026 01:40
@epompeii
epompeii force-pushed the u/ep/callback/metrics branch from a1ed80b to a932a18 Compare September 27, 2026 02:45
Every delivery attempt is counted by its class: a status class, a timeout, a connection error, or a block with its reason. Every callback that settles is recorded by its outcome and failure reason in histograms of its latency from the start of its delivery and of the attempts it made over its life. A gauge counts the callbacks waiting to be delivered, read from the database at each export on a read connection only if one is free, so a busy pool skips a point instead of stalling the export.

No attribute carries a URL, a header, an address, or any other value from the request.
@epompeii
epompeii force-pushed the u/ep/callback/metrics branch from a932a18 to 87646ad Compare September 27, 2026 03:03

This branch was successfully deployed

1 active (outdated) deployment
Cloudflare — 9312c58b Deployed Sep 26, 2026 by epompeii via Docker / Build API Docker (arm64) #1704
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant