Conversation
epompeii
marked this pull request as ready for review
September 26, 2026 02:38
Contributor
|
| Project | Bencher |
| Branch | u/ep/callback/github |
| Testbed | intel-v1 |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result microseconds (碌s) (Result 螖%) | Upper Boundary microseconds (碌s) (Limit %) |
|---|---|---|---|
| Adapter::Json | 馃搱 view plot 馃毞 view threshold | 5.18 碌s(+4.70%)Baseline: 4.95 碌s | 6.02 碌s (86.03%) |
| Adapter::Magic (JSON) | 馃搱 view plot 馃毞 view threshold | 4.97 碌s(+4.00%)Baseline: 4.78 碌s | 5.70 碌s (87.18%) |
| Adapter::Magic (Rust) | 馃搱 view plot 馃毞 view threshold | 27.97 碌s(+4.51%)Baseline: 26.76 碌s | 30.35 碌s (92.16%) |
| Adapter::Rust | 馃搱 view plot 馃毞 view threshold | 4.68 碌s(+13.86%)Baseline: 4.11 碌s | 6.42 碌s (72.93%) |
| Adapter::RustBench | 馃搱 view plot 馃毞 view threshold | 4.68 碌s(+14.03%)Baseline: 4.10 碌s | 6.39 碌s (73.23%) |
epompeii
added this pull request to stack #1054
September 26, 2026 02:44
epompeii
marked this pull request as draft
September 26, 2026 03:30
epompeii
marked this pull request as ready for review
September 26, 2026 04:19
epompeii
marked this pull request as draft
September 26, 2026 04:27
epompeii
force-pushed
the
u/ep/callback/github
branch
from
September 26, 2026 17:40
89c749b to
83ecc2b
Compare
epompeii
force-pushed
the
u/ep/callback/github
branch
2 times, most recently
from
September 26, 2026 22:04
6fd68fe to
e6ccb1b
Compare
With `--detach` and `--github-actions`, `bencher run` starts the check, composes a `repository_dispatch` callback authorized by the new `--ci-callback-token`, and posts nothing else. `--detach` with `--github-actions` requires `--ci-callback-token`, refused before the run starts, and `--callback-url` conflicts with `--github-actions`. The server sends the dispatch on every plan when it goes to github.com, and skips it without a paid plan only on GitHub Enterprise Server; a skipped callback completes the check as neutral. Both tokens are Secret, so a release build never prints them. An empty or blank `--ci-callback-token` is refused as it parses, with a message that says to check the secret, and so is an empty or blank `--github-actions` token. That is a breaking change: an empty `--github-actions` token used to be ignored outside GitHub Actions, to only warn on an event other than a pull request, and to fail a pull request's run after the results were uploaded. The dispatch's `client_payload` holds two objects. `bencher` holds the Job UUID and the Project slug, which the second workflow reads, and the run's `--ci-id`, pull request number, `--build-time`, and `--file-size`. `github` holds the head SHA the check was started at and the check. On a `repository_dispatch` event, `bencher run --job` reads the payload: it completes the check the detached run started, or creates one at its head SHA, and comments on its pull request under the same tag as the detached run, unless `--ci-number` or `--ci-id` is given. A remote job that fails or is canceled now completes its GitHub Check as a failure or as cancelled; it used to complete it as a success.
epompeii
force-pushed
the
u/ep/callback/github
branch
from
September 27, 2026 01:40
e6ccb1b to
34903a6
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Breaking:
bencher run --detach --github-actionsrequires the new--ci-callback-token, refused before the run starts, since nothing else would complete the check it starts. A workflow that detaches with--github-actionsand no callback token fails.--github-actionstoken, such as an unset secret, is refused as it parses, with a message that says to check the secret. A workflow that passes an unset secret fails before the run starts, off GitHub Actions and on events other than a pull request too.On GitHub Actions a bare metal run can submit its job without holding a runner for the length of the benchmark. Workflow 1 submits and exits; Bencher sends a
repository_dispatchwhen the job finishes; workflow 2 attaches and posts the results on workflow 1's check and pull request.Workflow 1:
--detachwith--github-actionsand--ci-callback-token.bencher runstarts the in-progress check, composes the callback, submits, prints the report, posts nothing else, and exits zero. A callback that fails to compose fails the check and submits nothing.repository_dispatchwith event typebencher_runto the repository'sdispatchesendpoint under the GitHub API URL, authorized by--ci-callback-token: a fine-grained personal access token with contents write on the one repository, since the workflow's own token has expired by the time the job finishes.client_payloadholds two snake_case objects, and an absent value has no key.bencherholds theprojectslug and thejobUUID as placeholders, which workflow 2 reads, and the run'sci_id, its pull request asci_number, and itsbuild_timeandfile_sizeflags.githubholds the headshathe check was started at and thecheck.--callback-urlconflicts with--github-actions. An empty or blank--ci-callback-tokenis refused as it parses, with the same message as--github-actions. Both tokens areSecret, so a release build never prints them.api.github.comon every plan, and a dispatch to GitHub Enterprise Server is an ordinary callback that needs a paid plan. A callback the server skips completes the check asneutralwith the notice. If the job cannot be read after submitting, the check stays in progress for workflow 2, with a warning.Workflow 2:
bencher run --jobon arepository_dispatchevent.client_payload: it completes the check workflow 1 started, or creates one at the payload's head SHA, and comments on the payload's pull request under the same tag as workflow 1, from itsci_idand flags. An explicit--ci-numberor--ci-idwins.Fix: a remote job that fails or is canceled completes its GitHub Check as
failureorcancelled, whatever its alerts, for an attachedbencher run --imageas well.No automated test runs three paths end to end, because the local stand-in for the GitHub API is
httpand a callback must behttps: the check id's path from the started check into the payload (each step on it is unit tested), theneutralcompletion of a skipped callback's check, and the warning when the job cannot be read back after submitting. An end-to-end run on GitHub Actions against the dev deployment covered the first, and the second before the dispatch toapi.github.comwent out on every plan.Needed for #1046