Skip to content

Hand a detached GitHub Actions run's check to its attach - #1063

Draft
epompeii wants to merge 1 commit into
u/ep/callback/clifrom
u/ep/callback/github
Draft

epompeii wants to merge 1 commit into
u/ep/callback/clifrom
u/ep/callback/github

Conversation

@epompeii

@epompeii epompeii commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Breaking:

  • bencher run --detach --github-actions requires the new --ci-callback-token, refused before the run starts, since nothing else would complete the check it starts. A workflow that detaches with --github-actions and no callback token fails.
  • An empty or blank --github-actions token, such as an unset secret, is refused as it parses, with a message that says to check the secret. A workflow that passes an unset secret fails before the run starts, off GitHub Actions and on events other than a pull request too.

On GitHub Actions a bare metal run can submit its job without holding a runner for the length of the benchmark. Workflow 1 submits and exits; Bencher sends a repository_dispatch when the job finishes; workflow 2 attaches and posts the results on workflow 1's check and pull request.

Workflow 1: --detach with --github-actions and --ci-callback-token.

  • bencher run starts the in-progress check, composes the callback, submits, prints the report, posts nothing else, and exits zero. A callback that fails to compose fails the check and submits nothing.
  • The callback is a repository_dispatch with event type bencher_run to the repository's dispatches endpoint under the GitHub API URL, authorized by --ci-callback-token: a fine-grained personal access token with contents write on the one repository, since the workflow's own token has expired by the time the job finishes.
  • The client_payload holds two snake_case objects, and an absent value has no key. bencher holds the project slug and the job UUID as placeholders, which workflow 2 reads, and the run's ci_id, its pull request as ci_number, and its build_time and file_size flags. github holds the head sha the check was started at and the check.
  • --callback-url conflicts with --github-actions. An empty or blank --ci-callback-token is refused as it parses, with the same message as --github-actions. Both tokens are Secret, so a release build never prints them.
  • The server sends the dispatch to api.github.com on every plan, and a dispatch to GitHub Enterprise Server is an ordinary callback that needs a paid plan. A callback the server skips completes the check as neutral with the notice. If the job cannot be read after submitting, the check stays in progress for workflow 2, with a warning.

Workflow 2: bencher run --job on a repository_dispatch event.

  • It reads client_payload: it completes the check workflow 1 started, or creates one at the payload's head SHA, and comments on the payload's pull request under the same tag as workflow 1, from its ci_id and flags. An explicit --ci-number or --ci-id wins.
  • The payload's check is taken before the version and safety checks, so an error there still fails it.
  • A malformed payload is ignored with a warning, and the attach goes on as on any other event.

Fix: a remote job that fails or is canceled completes its GitHub Check as failure or cancelled, whatever its alerts, for an attached bencher run --image as well.

No automated test runs three paths end to end, because the local stand-in for the GitHub API is http and a callback must be https: the check id's path from the started check into the payload (each step on it is unit tested), the neutral completion of a skipped callback's check, and the warning when the job cannot be read back after submitting. An end-to-end run on GitHub Actions against the dev deployment covered the first, and the second before the dispatch to api.github.com went out on every plan.

Needed for #1046

@epompeii
epompeii marked this pull request as ready for review September 26, 2026 02:38
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

馃惏 Bencher Report

ProjectBencher
Branchu/ep/callback/github
Testbedintel-v1
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
microseconds (碌s)
(Result 螖%)
Upper Boundary
microseconds (碌s)
(Limit %)
Adapter::Json馃搱 view plot
馃毞 view threshold
5.18 碌s
(+4.70%)Baseline: 4.95 碌s
6.02 碌s
(86.03%)
Adapter::Magic (JSON)馃搱 view plot
馃毞 view threshold
4.97 碌s
(+4.00%)Baseline: 4.78 碌s
5.70 碌s
(87.18%)
Adapter::Magic (Rust)馃搱 view plot
馃毞 view threshold
27.97 碌s
(+4.51%)Baseline: 26.76 碌s
30.35 碌s
(92.16%)
Adapter::Rust馃搱 view plot
馃毞 view threshold
4.68 碌s
(+13.86%)Baseline: 4.11 碌s
6.42 碌s
(72.93%)
Adapter::RustBench馃搱 view plot
馃毞 view threshold
4.68 碌s
(+14.03%)Baseline: 4.10 碌s
6.39 碌s
(73.23%)
馃惏 View full continuous benchmarking report in Bencher

@epompeii
epompeii added this pull request to stack #1054 September 26, 2026 02:44
@epompeii
epompeii marked this pull request as draft September 26, 2026 03:30
@epompeii
epompeii marked this pull request as ready for review September 26, 2026 04:19
@epompeii
epompeii marked this pull request as draft September 26, 2026 04:27
@epompeii
epompeii force-pushed the u/ep/callback/github branch from 89c749b to 83ecc2b Compare September 26, 2026 17:40
@epompeii
epompeii force-pushed the u/ep/callback/github branch 2 times, most recently from 6fd68fe to e6ccb1b Compare September 26, 2026 22:04
With `--detach` and `--github-actions`, `bencher run` starts the check, composes a `repository_dispatch` callback authorized by the new `--ci-callback-token`, and posts nothing else. `--detach` with `--github-actions` requires `--ci-callback-token`, refused before the run starts, and `--callback-url` conflicts with `--github-actions`. The server sends the dispatch on every plan when it goes to github.com, and skips it without a paid plan only on GitHub Enterprise Server; a skipped callback completes the check as neutral. Both tokens are Secret, so a release build never prints them. An empty or blank `--ci-callback-token` is refused as it parses, with a message that says to check the secret, and so is an empty or blank `--github-actions` token. That is a breaking change: an empty `--github-actions` token used to be ignored outside GitHub Actions, to only warn on an event other than a pull request, and to fail a pull request's run after the results were uploaded.

The dispatch's `client_payload` holds two objects. `bencher` holds the Job UUID and the Project slug, which the second workflow reads, and the run's `--ci-id`, pull request number, `--build-time`, and `--file-size`. `github` holds the head SHA the check was started at and the check. On a `repository_dispatch` event, `bencher run --job` reads the payload: it completes the check the detached run started, or creates one at its head SHA, and comments on its pull request under the same tag as the detached run, unless `--ci-number` or `--ci-id` is given. A remote job that fails or is canceled now completes its GitHub Check as a failure or as cancelled; it used to complete it as a success.
@epompeii
epompeii force-pushed the u/ep/callback/github branch from e6ccb1b to 34903a6 Compare September 27, 2026 01:40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant