Fire job callbacks once a job reaches a terminal state

[epompeii]

A callback is sent once, after its job reaches a terminal state, however many paths finish the job.

• The fire. Every write that makes a job processed, failed, or canceled (nine sites across the runner channel, the heartbeat timeout, and the reprocessing of completed jobs) fires the job's callback once its update commits, which hands it to a delivery task. A job has at most one delivery task at a time, tracked in memory, and the task sends only a pending callback of a terminal job, so a second fire, a skipped callback, or no callback sends nothing. completed never fires.
• The delivery. The task opens the sealed request, which validates again, and renders its body with the job's values. It builds the job's report, exactly as the report endpoint returns it, only for a callback without a body or one whose body sends {{ report }}, on a read connection so a large report never holds the writer, and at most two at a time, so a burst of callbacks cannot take every worker thread. Without a free read connection, it tries again each pool timeout. A request that does not open or does not validate, or whose report does not build, is never sent and fails.
• The attempts. Up to three per callback, 4 s and 16 s apart. A timeout, a connection error, 408, 429, and any 5xx retry; any other status and a blocked address are final. Each attempt, and the settle of the one that ends the delivery, is one update filtered on pending, and a delivery whose write finds nothing pending stops. The count lives on the row, so a delivery started again continues from it.
• Logs. One line per attempt with the job, organization, project, destination host, attempt number, outcome, status, and duration, plus the refused address of a block and the cause chain of a connection error, whose URL is stripped. Never a header value, the URL beyond its host, or the body.
• Ownership. The API context owns every delivery task. Shutdown cuts a delivery short, including its wait for a report build or a read connection, without a write, so its callback stays pending, and waits for every task once the rate limiting state is saved.
• Startup. The server builds the callback client at boot: if the build fails, it logs the cause once and starts anyway, and every attempt is then a connection error. Job recovery, which runs once the server accepts connections, ends by firing every pending callback on a terminal job, even when its query for in-flight jobs fails.

Nothing creates a callback yet. The tests inject a scripted or recording sender; no test and no configuration reaches the network.

Needed for #1046

[github-actions]

Bencher Report

Project	Bencher
Branch	u/ep/callback/fire
Testbed	intel-v1

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
Adapter::Json	📈 view plot 🚷 view threshold	5.18 µs (+5.25%) Baseline: 4.92 µs	5.99 µs (86.43%)
Adapter::Magic (JSON)	📈 view plot 🚷 view threshold	4.97 µs (+4.60%) Baseline: 4.75 µs	5.68 µs (87.63%)
Adapter::Magic (Rust)	📈 view plot 🚷 view threshold	27.53 µs (+3.31%) Baseline: 26.64 µs	30.29 µs (90.87%)
Adapter::Rust	📈 view plot 🚷 view threshold	4.66 µs (+15.47%) Baseline: 4.04 µs	6.34 µs (73.60%)
Adapter::RustBench	📈 view plot 🚷 view threshold	4.64 µs (+15.19%) Baseline: 4.03 µs	6.31 µs (73.57%)

🐰 View full continuous benchmarking report in Bencher