docs(merge-queue): document the per-state duration metric - #339
Merged
Merged
Conversation
trunk-io/trunk#33921 adds `mq_pr_state_duration_1h_seconds` to /v1/getMergeQueueMetrics: a histogram plus `_p{50,75,90,95,99}` gauges, labelled by `state`, covering every queue state rather than just the two that `mq_pr_wait_duration_1h_seconds` and `mq_pr_test_duration_1h_seconds` already cover. Adds it to the rolling-window metric table, plus a Per-state durations section covering the `state` label values, the two behaviors that surprise people (terminal states measure time-until-resubmit, and a state series is absent rather than zero when nothing left that state), example PromQL, and sample exposition. Also documents the `_p50`/`_p75`/`_p90`/`_p95`/`_p99` companion gauges, which the endpoint has always emitted for every duration histogram but which the page never mentioned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
A reader-perspective review of the page found four defects in the
material added by the previous commit, plus one pre-existing error.
The alert example could not fire when it mattered. Every duration here is
recorded when a PR *leaves* a state, so during an active stall nothing
exits and the series thins out or vanishes — the `pending_failure` p95
alert would only have fired after recovery. Replaced it with a note that
this family is retrospective, and that stall detection belongs on the
point-in-time gauges.
The `histogram_quantile` example aggregated `by (le)`, which drops the
`state` label the panel is supposed to break out by; now `by (le, state)`.
The "where is the time going" example excluded only the terminal states,
which silently mixed pre-queue `not_ready` time into a breakdown of
in-queue time. Time in queue is exactly pending + testing + tests_passed +
pending_failure, so the examples and the Warning now select those four by
name. That also fixes a contradiction: an open label set means a deny-list
regex picks up whatever state is added next.
"Carrying the same labels as the histogram" implied `_p95{le=...}`; the
percentile gauges carry every label except `le`. Also documented that they
are computed over raw observations rather than buckets — which is the real
argument for them, since a bucket-derived quantile past the 7200s top
boundary is extrapolation — and that they cannot be re-aggregated across
repos.
Added which metric to prefer for new dashboards, since the page documented
two ways to get pending duration and recommended neither.
Pre-existing, corrected against the strings the endpoint emits: the sample
output gave `mq_testing_slots_active` the HELP text of `mq_prs_testing`
("PRs currently in TESTING state"), which are different numbers whenever
batching is on. Added the missing `mq_prs_testing` sample alongside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Validated every example on the page against a local Prometheus fed by a real staging scrape (api.trunk-staging.io, 9 repos) plus a synthetic mq_pr_state_duration_1h_seconds built to the schema in trunk-io/trunk#33921, which is not deployed to staging yet. Three pre-existing examples returned zero series — in Grafana they render as "No data", which is why nobody noticed. Failure rate: `ignoring(conclusion) sum(...)` cannot match. A bare `sum()` strips every label, so the left side keeps repo/branch/queue_type and the right side has none. Now sums `without (conclusion)` to keep a per-repo denominator, and restricts it to the three exit conclusions — this metric also carries non-terminal `conclusion` values, so the old denominator counted PENDING and TESTING transitions as part of the failure-rate base. Returns 2 series with plausible values where it returned nothing. Restart ratio: `conclusion` exists only on the right-hand vector, so the two never match. Added `ignoring(conclusion)`; returns 4 series. P90 wait: `sum(...) by (le)` drops `repo`, silently blending every scraped repo into one number — and the page recommends scraping a whole org. Now `by (le, repo)`. The five per-state examples added earlier all evaluate correctly, including the group_left share-of-total (the four shares sum to 1.0) and the `ignoring(state)` restart ratio. Two controls back the guidance: a p95 over all states returns 7200 (pinned to the top bucket by the terminal states) against 5400 for the in-queue states alone, and the `by (le)` form the earlier commit removed does collapse all four states into a single line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Docs PR opened: #340 Added a changelog entry announcing the new Merge Queue per-state duration Prometheus metric and its companion percentile gauges. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents
mq_pr_state_duration_1h_seconds, which trunk-io/trunk#33921 adds to/v1/getMergeQueueMetrics.What changed
The metric itself. A row in the rolling-window metrics table, plus a new Per-state durations section.
mq_pr_wait_duration_1h_secondsandmq_pr_test_duration_1h_secondseach measure exactly one state; the new metric measures every state a PR passes through, with the state on astatelabel.The section documents the label values (
not_ready,pending,testing,tests_passed,pending_failure, and the three terminal states) and the two behaviors that would otherwise bite someone writing their first query against it:merged/failed/cancelledare only ever left by a re-submit, so those series are time-until-resubmit — days, landing in+Inf. Aggregating across states without filtering them out wrecks the quantile, so the section says to filter and the example queries do.stateseries only exists once some PR actually exited that state in the window, which behaves differently in PromQL from the zero-filled histogram the older wait/test metrics emit. It also notes explicitly that those two metrics are unchanged and are not going away, since this endpoint is scraped by customer Prometheus configs.Percentile gauges.
_p50/_p75/_p90/_p95/_p99companion gauges are documented for the first time. The endpoint has emitted them for every duration histogram since before this PR — the page only ever mentioned_bucket/_sum/_count. #33921 adds them for the new metric too, and describing it accurately meant describing them.Examples. A per-state PromQL block (percentile gauge read directly, mean-per-state with terminal states excluded, a
pending_failurealert,histogram_quantileover one state's buckets) and a sample exposition block. The sample# HELPlines are copied verbatim from the strings in #33921; the intermediate buckets are elided with a comment rather than silently dropped.Notes
openapi.jsondescribes the endpoint as an opaquetext/plainbody and doesn't enumerate metric names, so nothing to update there.trunk checkwants to reformat the whole file (pre-existing bullet-style issues onmain), which I deliberately left out to keep this reviewable.🤖 Generated with Claude Code