Publish benchmark results in the style of the sibling LinksPlatform comparisons

[konard]

Summary

Brings the benchmarking CI/CD in line with Comparisons.PostgreSQLVSDoublets and Comparisons.Neo4jVSDoublets, so the benchmark results are shown, documented and recalculated automatically — and are readable in the repository without a local benchmark run.

Fixes #14

The benchmark CI was red, so nothing could be published

Rust Benchmark had been failing on every branch, including main, since 2026-04-14. Root cause: the workflow and rust/rust-toolchain.toml both asked for a rolling nightly, and two dependencies rely on unstable APIs that changed underneath the repository:

error[E0512]: cannot transmute between types of different sizes
  --> ethnum-1.5.2/src/error.rs:16

error[E0277]: the trait bound `Flow: Residual<()>` is not satisfied
  --> doublets-patched/dev-deps/data-rs/src/flow.rs:16

(run 35022182087, reproduced locally.)

ethnum relies on the layout of core::num::IntErrorKind; the patched doublets dev dependency implements the nightly-only Try/Residual traits. The toolchain is now pinned to nightly-2026-04-14 — the last nightly the benchmark built with — so it is updated deliberately instead of breaking silently.

Two more CI defects found on the way:

• cargo bench ... | tee out.txt had no set -o pipefail, so a crashing benchmark produced a green run and an empty out.txt.
• The benchmark jobs installed the toolchain without rustfmt/clippy, which rust/rust-toolchain.toml requires, so rustup added them lazily mid-job and failed with detected conflict: 'bin/cargo-fmt'.

What the results pipeline now does

rust/out.py was rewritten. It previously knew about only two of the five benchmarked backends — the two NonVolatile Doublets variants never appeared anywhere — and it produced charts but no table, which is why the README results placeholder was never filled in.

It now:

• parses all 7 operations × 5 backends from criterion bencher output;
• writes rust/results.md, annotating every Doublets cell against the SpacetimeDB baseline (12.3x faster / 1.2x slower — slower is reported honestly);
• emits a provenance line (when, which Actions run, at which scale);
• replaces the README section between <!--BENCHMARK_RESULTS_START--> / <!--BENCHMARK_RESULTS_END-->;
• generates the linear and logarithmic charts and copies them into Docs/.

The workflow publishes all of that: the full run on main commits Docs/, README.md and rust/results.md as LinksPlatformBencher, exactly like the sibling repositories, and both the PR and full runs write the results table into the GitHub Actions job summary.

Tests

rust/test_out.py — 21 unit tests covering parsing (including thousand separators, missing files and noise lines), speedup formatting, the table, provenance, README replacement (including regex-unsafe table content such as \g<0>) and the CLI entry point.

A new results-pipeline CI job runs them and gates both benchmark jobs, so a 3 hour benchmark can no longer finish only to fail while publishing its results.

cd rust && python3 -m unittest test_out -v
# Ran 21 tests — OK

Verified locally

• cargo clippy --all-targets and cargo fmt --all -- --check pass on the pinned toolchain
• the SpacetimeDB WASM module builds and publishes
• a real benchmark run against a local SpacetimeDB 2.10.1 server produces the table and both charts

Also

• removed rust/rust_out, a 4.2 MB compiled binary committed by accident
• dropped the stale top-level spacetime-module/** path filter (the module lives in rust/spacetime-module)
• README: reference-style Results section with both charts served from Docs/ via permanent raw links, plus a Conclusion explaining the measured gap

[konard]

Working session summary

Root cause of the last CI failure found and fixed. Status while the long-running jobs finish:

What broke run 35028280108: Criterion prints its own errors to stdout (criterion-0.3.6/src/macros_private.rs:36), and it does so between the two halves of a bencher record (print!("test {} ... ") at src/report.rs:740, println!("bench: ...") at :756). A partially restored target/criterion/<id>/<size>/base directory made every record look like:

test create/SpacetimeDB/10 ... Criterion.rs ERROR: error: Failed to access file ".../base/sample.json": No such file or directory (os error 2)
bench:    24992765 ns/iter (+/- 1790323177)

so the single-line regex in out.py matched nothing → No benchmark data found in out.txt → exit 1. Verified by replaying the CI text through the old parser ({}) and the new one (both values recovered): experiments/reproduce-criterion-error-interleaving.py.

Fixes pushed (6af08a0, a50fe91, c9dacb0):

• rust/out.py parses the whole file with a pattern that tolerates interleaved output, without borrowing the next benchmark's number; prints the tail of unparsable input on failure.
• Both benchmark steps `rm -rf target/cri

Rest of the working session summary (586 characters)

terion` first, so stale comparison state can't pollute stdout.
- Artifacts upload with `if: always()`, so a failed results step still preserves `out.txt`.
- 3 new regression tests (26 total, all passing locally).
- Charts moved from `Docs/` to `docs/benchmarks/` — the repo already has a lowercase `docs/`, and the two would collide on the case-insensitive filesystem the macOS job checks out on.

Still running: CI on the new commits, and the full-scale local benchmark (5/35 measurements; the SpacetimeDB groups need ~16 min each) that will seed the committed charts and README table.

---

This summary was automatically extracted from the AI working session output.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $12.549547

📊 Context and tokens usage:

Claude Opus 5: (3 sub-sessions)

1. 117.0K / 1M (12%) input tokens, 29.8K / 128K (23%) output tokens
2. 115.0K / 1M (12%) input tokens, 34.4K / 128K (27%) output tokens
3. 92.3K / 1M (9%) input tokens, 24.6K / 128K (19%) output tokens

Total: (5.9K new + 293.5K cache writes + 13.7M cache reads) input tokens, 109.5K output tokens, $12.549547 cost

🤖 Models used:

• Tool: Anthropic Claude Code
• Requested: opus (claude-opus-5)
• Thinking level: high (~23999 tokens)
• Model: Claude Opus 5 (claude-opus-5)

📎 Log file uploaded as Gist (5652KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

🎉 Auto-merged

This pull request has been automatically merged by hive-mind.

• All CI checks have passed

---

Auto-merged by hive-mind with --auto-merge flag