Conversation
epompeii
added this pull request to stack #1054
September 26, 2026 02:44
epompeii
marked this pull request as ready for review
September 26, 2026 03:16
Contributor
|
Bencher Report| Project | Bencher |
| Branch | u/ep/callback/smoke |
| Testbed | intel-v1 |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | Latency | microseconds (µs) |
|---|---|---|
| Adapter::Json | 📈 view plot | 5.16 µs |
| Adapter::Magic (JSON) | 📈 view plot | 4.99 µs |
| Adapter::Magic (Rust) | 📈 view plot | 27.54 µs |
| Adapter::Rust | 📈 view plot | 4.70 µs |
| Adapter::RustBench | 📈 view plot | 4.65 µs |
epompeii
marked this pull request as draft
September 26, 2026 03:30
epompeii
marked this pull request as ready for review
September 26, 2026 04:19
epompeii
marked this pull request as draft
September 26, 2026 04:27
The runner smoke test now submits two detached jobs with a callback. On an organization without a plan, the CLI prints the skip notice and the processed job shows the callback skipped. On a self-hosted server, the organization is then licensed at Enterprise with the committed test key, and a callback to the GitHub repository_dispatch endpoint without a token settles failed with the 4xx GitHub answers. That covers the real send path against a public https receiver: seal, claim, the report body, address policy, DNS, TLS, and the recorded attempt. The license is removed afterwards. Bencher Cloud gives a paid plan only through Stripe, so the dev smoke test runs the skipped step only.
epompeii
force-pushed
the
u/ep/callback/smoke
branch
from
September 26, 2026 17:40
bd78eb3 to
4a0b1df
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The runner smoke test now sends a real callback, through the real sender and the real address policy, with no policy relaxed.
bencher run --image ... --callback-urlprintscallback skipped: requires a Bencher Plus plan, and once the job finishes its callback readsskippedwith no status.smoke ci. The smoke organization gets an Enterprise license signed in process with the committed test key, which only a debug build trusts, and the license is removed afterwards. A detached run with one harmless header sends its callback to GitHub'srepository_dispatchendpoint without a token. The step waits for the callback to settle and assertsfailedwith a 4xx: GitHub answers 401, or 403 or 429 under its rate limit for shared runner addresses. That one assertion covers the seal, the claim, the default report body, the address policy, DNS, TLS, the attempt record, and the settle.smoke dev. A release build trusts only the live license key, and dev has no paid plan without Stripe, so dev runs only the skipped step.The printout check on stdout runs only in a release build, since a debug build prints the callback in full. Every failure message carries the job's callback state, because CI discards the API's database with the run.
Needed for #1046