TeleMath, cost-metric generalisation, and the Stage 2 cascade - #1
Merged
Conversation
Measure end-to-end latency (E2EL), request throughput, and truncation
rate alongside TPOT in `cre evaluate`, and allow routing to be fit on
either metric with `cre fit --cost-metric {tpot,e2el}` (default tpot,
which reproduces the published results). E2EL is the informative cost
when pool members differ in output length rather than decode speed,
where TPOT cannot tell them apart.
Add `cascade_system_metrics` and a `cre cascade` command that compose
Stage 1+2 system TPOT (query-weighted, charging an escalated query's
discarded efficient pass per delivered token) and E2EL (the per-request
sum of both passes) from per-cluster measurements and the escalation rate.
Add the two per-cluster test-split stat files (configs/*_cascade_test.json)
so `cre cascade` reproduces the paper's Stage 1+2 latency with no GPU, and
document it in REPRODUCE.
Add TeleMath (netop/TeleMath) as an evaluation task. Its answers are numerical, so they are matched by relative tolerance rather than exact equality. Separate arms cover thinking and non-thinking models, plus a pre-rendered Gemma 4 variant whose thinking switch is a chat-template kwarg baked into the prompt text at prep time, since vLLM's benchmark loader never forwards template kwargs. Record per-question outcomes (qid, cluster, run, correct, output_len) and, behind a flag, full generations with token counts. Both are irrecoverable once a run finishes, and are needed for confidence intervals and for building Stage 2 QE training data. A stable question_id lets outcomes join across models and runs. Bound the TeleMath answer search to the output tail and use a number pattern with no adjacent variable-length digit runs, keeping parsing linear in the window. Move split_thinking into cre_router.textutils, now shared by the answer parsers and the QE input formatter.
Cascade cost now takes a full escalation chain per cluster, ordered from the Stage 1 model up to the strongest tier, with a reach count per tier. A cluster can therefore route direct, gate once, or gate repeatedly. The two-tier cascade_system_metrics becomes a thin wrapper over it, so the published 9.7 ms AIME and 23.8 ms TeleQnA figures serve as the N=2 regression test. Accuracy composes per query at a single gate: an accepted output keeps the efficient model's correctness, an escalated one takes the strong model's. It remains two-tier. Extending it to walk a longer chain is not done here. Record the measured per-cluster cascade accuracy in the AIME and TeleQnA cascade configs, so cre cascade reproduces the reported system accuracy of 88.4% and 74.3% rather than cost alone.
Add a qe-cascade command that replays a trained quality-estimation classifier over saved generations, decides accept or escalate per query, and reports the resulting per-cluster cascade accuracy and escalation counts. Those feed the Stage 1+2 cost and accuracy functions. Add prep_qe.py to build QE training data from the generations JSONL that cre evaluate now writes, and wire --save-generations through the CLI. Restrict the QE dataset loader to train* and test* files. It previously globbed every JSONL in the directory, which pulled a cascade-test sidecar into the test split.
The C0 and C1 error rates were transposed relative to the measurement sheet, which records 0.293 on C0 and 0.332 on C1. Routing is unaffected. Gemma4-E4B lies inside the convex hull under either ordering, with margins of 0.039 and 0.052 before and 0.078 and 0.013 after, so it is never the argmin at any lambda. lambda*, the assignment and every reported accuracy are unchanged. (cherry picked from commit 80a7b68)
Give the measurement conditions (2x A100 SXM 80 GB, Python 3.11, 32 concurrent requests, 5-run mean) alongside the pinned versions, and refer to the pinned file rather than repeating version numbers in prose, so the two cannot drift apart.
Both qe-cascade and the qe evaluate script defaulted --base-tokenizer to a Hub model id, which overrode QEClassifier's own fallback and made either command fail on a machine without internet, even though every checkpoint we train ships its tokenizer alongside the weights. Default to None so the fallback applies; a base model id is now only needed for a checkpoint saved without a tokenizer.
Add the command chain from a run's own generations to a cascade config: capture generations during evaluation, convert them to the schema qe-train reads, then replay the classifier over the gated clusters.
Add a GitHub Actions workflow that installs the package and runs pytest on Python 3.10 and 3.12, the ends of the supported range. The suite needs no GPU and no optional extras.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the cost-metric and Stage 2 work onto
main. Nine commits, 23 files.Evaluation
netop/TeleMath) as a task: numerical answers matched by relative tolerance, separate thinking and non-thinking arms, and a pre-rendered Gemma 4 variant whose thinking switch is a chat-template kwarg the benchmark loader will not forward.--save-generationsadds full outputs. Both are irrecoverable once a run finishes and are required for confidence intervals and for building QE data. A stablequestion_idjoins outcomes across models and runs.Routing
--cost-metric e2el, needed whenever pool members differ in output length rather than decode speed.Stage 2
cre qe-cascadereplays a trained classifier over saved generations and writes per-cluster cascade accuracy and escalation counts into a cascade config forcre cascade.prep_qe.pybuilds QE training data from the generations JSONL.cre qe-train --dataset <dir>now reads onlytrain*andtest*from the directory, so the differently-shaped generations files kept alongside are not loaded as data.Testing: 163 tests, 3 skipped, green on Python 3.10 and 3.12.