Bradley/stackbench test by bradleyshep · Pull Request #5766 · clockworklabs/SpacetimeDB

bradleyshep · 2026-08-19T17:14:23Z

Description of Changes

This PR adds Stack Bench, a reproducible benchmark for coding agents building applications against SpacetimeDB, PostgreSQL, and MongoDB.

Benchmark definitions and grading

Adds versioned tracks, levels, feature packs, specification packs, recipes, prompts, checks, and dependency declarations.
Supports selecting complete recipes, individual packs, or individual checks.
Separates what the agent is asked to build from what the finished application is evaluated against.
Adds typed scenario actions for browser interaction, authenticated server calls, concurrent operations, direct database writes, reconnects, reloads, and service lifecycle operations.
Records each check as passed, failed, inconclusive, or harness_failure.
Keeps the declared score denominator fixed when evidence is missing or inconclusive.
Adds reference, mutation, and null-control qualification:
- reference applications for all three stacks;
- source-bound mutation manifests;
- exact mutation-target and collateral-failure validation;
- blank-application controls;
- versioned calibrations and promotion catalogs.
Adds immutable identities for recipes, prompts, fixtures, mutations, checks, runtime images, adapters, and evidence.

Ecommerce benchmark

Adds ecommerce L1 coverage for accounts, sessions, storefronts, carts, checkout, purchases, warehouses, reviews, authorization, accounting, concurrency, live updates, persistence, reconnects, and direct database writes.
Adds ecommerce L2 coverage for fulfilment, shipping, warehouse transfers, cancellations, returns, historical pricing, operational views, recommendations, authorization, stock conservation, and refund accounting.
Adds complete ecommerce reference applications for MongoDB, PostgreSQL, and SpacetimeDB.
Promotes qualified ecommerce L1 modular 2.4.
Keeps ecommerce L2 1.4 as the promoted L2 release.
Adds L2 1.5 as a draft cumulative successor based on L1 2.4:
- 76 checks;
- 117 available points;
- static mutation coverage for all 74 scored checks on every supported stack;
- candidate reference fixtures, mutation manifests, and calibration;
- live qualification still required before promotion.

Runs, repairs, and campaigns

Adds bounded correction rounds with rollback when a repair loses previously established evidence.
Stops cumulative execution when a lower level remains incomplete.
Adds source checkpoints and operator-granted continuation rounds without rewriting the original result.
Adds campaign manifests containing exact stacks, repetitions, parallelism, recipes, models, pricing, prompt treatments, and correction budgets.
Supports concurrent attempts of the same or different stacks using isolated ports, databases, modules, containers, work directories, leases, and result directories.
Adds durable campaign scheduling, retries, locking, interruption recovery, reconciliation, and cleanup.
Preserves invalid and interrupted attempts instead of presenting them as comparison data.
Adds reproducible JSON and HTML reports with scores, coverage, cost, tokens, elapsed time, correction rounds, raw evidence, screenshots, videos, traces, and prompts.

CLI, dashboard, and appliance

Adds Stack Bench CLI commands for runs, campaigns, preflight, qualification, composition, repair continuations, recovery, release verification, and reporting.
Adds a Dockerized local dashboard that reads and controls the same durable campaign state as the CLI.
Adds a Linux/amd64 Docker appliance with pinned controller and build images, preflight checks, resource leases, reference qualification, release identity verification, and recovery tooling.
Adds model-free reference adapters and campaign trials for testing orchestration without provider spend.
Adds retained grader stdout/stderr and structured browser-initialization evidence.
Hardens process cleanup, destructive filesystem operations, campaign locks, recovery authority, credential redaction, and failed teardown handling.
Prevents local notes, generated reports, media, runtime state, and mutation scratch files from entering controller image builds.

SpacetimeDB TypeScript SDK

Changes exported TableRef from an anonymous readonly type alias to a readonly interface with the same members.
Produces shorter, named TypeScript diagnostics when a TableRef is used incorrectly.
Changes automatic ConnectionManager reconnects to reuse the identity token previously issued by the server.
Keeps explicit connection rebuilds bound to the token supplied by the caller.
Adds unit tests for reconnect token reuse and TableRef diagnostics.
Adds an opt-in live-host reconnect identity test.

SpacetimeDB CLI

Changes spacetime dev to retain configuration changes made during direct-login publishing.
Reuses that authenticated configuration for log streaming and subsequent rebuilds.

SpacetimeDB skills

Expands the TypeScript, Rust, and C# server skills with examples and guidance for client visibility filters, procedures, HTTP endpoints, indexes, and uniqueness constraints.
Updates the TypeScript client skill’s authentication and reconnect guidance.

API and ABI breaking changes

This PR changes the public Rust signature of:

spacetimedb_cli::publish::exec_from_entry

The function now accepts &mut Config instead of taking Config by value. External Rust callers must pass a mutable reference. The API-breaking label should be applied unless the previous entry point is preserved through a compatibility wrapper.

The exported TypeScript TableRef declaration changes from a type alias to an interface while retaining the same readonly member contract.

Automatic TypeScript client reconnection also changes behavior by presenting the identity token from the previous successful connection.

Expected complexity level and risk

3 / 5

Most of the implementation is isolated under tools/stack-bench. The principal risks are benchmark correctness, evidence integrity, concurrent resource isolation, interrupted-run recovery, Docker cleanup, and credential handling.

The TypeScript reconnect change and spacetimedb-cli helper signature are the only changes outside the benchmark and skill directories.

Testing

The two fake storefronts said nothing a screenshot could not. They are now the system itself. The harness slide draws what runs: controller, preflight gate, agent adapter and backend lease; the sealed container and per-run database services beneath them; the deployed app; the grader with one browser context per actor and its capability-scoped actions; and the split into typed verdicts and the evidence bundle. Flow animates down the live edges, and a dashed return edge carries failures back to the agent, labelled behavioural. The contention slide is a transaction timeline: alice and bob both read stock=1 before either writes, alice commits, bob is rejected. A playhead sweeps the time axis. That is the moment the level-3 criteria exist to force.

The bet is now one criterion and three implementations of it side by side: a reducer that is one transaction by construction, a SQL transaction that has to remember FOR UPDATE, and a findOneAndUpdate whose atomicity depends on the guard living inside the query and on staying within one document. Same criterion, same prompt; what differs is how much the agent must get right. You can't game it shows the planted defect as a diff, the guard line removed from the reference app, with the expectations beneath it: that exact criterion must fail, everything else must still pass, a blank build must score zero. Receipts shows the artifact tree of a run and a real verdict object, with its typed state, what was observed, the deadline, the evidence paths and the exact recipe it was graded under.

Nobody reads code on a slide. The bet is now three operation chains that build left to right: one pill for a stack that gives you a transaction, five for one that makes you remember the row lock, three for one that is atomic only while the guard sits inside the query and the write stays in a single document. The count of things the agent must get right is the point. The contention slide became a simulation of the outcome rather than a transaction trace: the same collision runs on two stacks side by side, both requests crossing at once, and the stock lands on sold once or sold twice. The mutation slide shows the guard dissolving out of the checkout flow, and the verdict is a record card instead of a JSON blob. The run log and file tree stay, since those are what an operator actually reads.

Two animations running at once meant neither could be watched. The contention slide is now a single collision — both checkouts crossing one rail set against one unit of stock — and a single outcome bar beneath it that resolves, first to the stack that serialises the write, then to the one where the guard was missed. Same comparison, read in sequence instead of in parallel. Receipts drops the file tree beside the verdict and shows one opened file: its path, the failed criterion, what was observed, the actor, the deadline, the evidence and the recipe. The rest of the bundle is named in a line underneath rather than drawn as a competing picture.

The swap slide claimed a database was swapping in and then nothing changed. The bill now tracks the socket: database a reads 51/51, $4.73, 1 round; b reads $7.10 and 2; c drops to 49/51, $11.28 and 3. Swap the database, watch the numbers move. Both animations tile the same nine second cycle so the label and the bill are never out of step. The run slide was a log pretending to be a loop. It is now the cycle itself: build, grade, all pass?, and on failure a bug report and a fix round returning to grade. Nodes light as the run reaches them and a state row reads round, score, spend and outcome, going 48/51 and three failed on the first pass, then 51/51 and accepted on the second. Each value holds an explicit window so the readout never contradicts the stage. The requirement-first framing moved out of slide two, which confused the deck by naming criteria before the deck explains them. Slide two is now the setup; where the guarantee lives comes after the oversell scenario, as its explanation.

Rewrote every construction that defined something by what it lacks. The hero now names Stack Bench as the thing that measures agent-built backends. The loop headline reads that the first attempt is only the first round. The database serialises the write itself and keeps the count; the other side offers the pieces and leaves the rule to you. A criterion earns the right to score once it kills a planted defect cleanly and holds a blank build at zero, and a survivor sends that criterion back for repair. Receipts states that every published score traces back to its files.

Two noun phrases followed by a verb clause read as a mistake, so the headline is three noun phrases: same app, same model, different database. The scoreboard title matches: one store, three databases, three bills. Either/or now carries a verb on both sides, keeps the count or keeps it, and the socket label drops the imperative for a statement. The deck also called the variable a stack in the copy and a database in the headlines. It is a database throughout now; Stack Bench stays the product name.

The contention slide showed one attack and asserted the rest. It is now a picker: six named attacks down the side, each replaying its own sequence in the stage. Concurrent checkout, a backend restart mid-order, a client that drops and returns, cart isolation between shoppers, a write forged to look like someone else, and two writers racing the same catalogue. Selecting one restarts its steps and lands on its verdict, so the breadth is visible rather than claimed. Also restores the expectation-row styles on the grader slide, which an earlier cleanup removed and left running together as one line.

Restores the version from cfa813f, before the animated demos replaced the written slides. Nine slides of plain prose and lists covering the same ground: the question, the loop, verification, tracks and levels, grader validation, reproducibility, evidence, and results. Useful where motion is a liability, on a projector, in a PDF, or for a reader who wants the argument in text. Normalised to the single accent so both documents look like the same product.

…m, trim the plain doc Every run ends at a full score because the loop continues until each criterion passes, so the scoreboard reads 51/51 across the board and the difference lives in spend and rounds. The mid-run readout still shows 48/51 during the first grading pass, which is what the agent is being told at that moment. The what-is-measured slide dropped its rig. It sets up the method rather than demonstrating anything, and the comparison already lands on the scoreboard, so it is a headline and two sentences now. The plain document is rewritten: static, ecommerce only, and down to the parts that carry weight. What is measured, how the loop runs, the six attacks, where the guarantee lives, how the grader is proven, and the bill. The architecture inventory, the axis taxonomy and the reproducibility checklist are gone. The hero also stops presenting agent-written backends as news.

…agram track the run The what-is-measured slide gets three rails that advance toward one finish line. Every database lands on 51 / 51, and the round count and the cost show up as each one arrives, so the slide states the measurement in the shape of the picture. The loop diagram now walks the run. Grade and the decision are reached twice per pass, which a shared keyframe plus an offset could not express, so each node carries its own windows across the 14s cycle. Two edges used to pulse green permanently while the rest sat still; each edge now lights only while the run is crossing it, and the branch labels appear with the decision that takes them. The readout holds a value in every cell at every point in the cycle: score moves through pending, 48 / 51 and 51 / 51, and outcome through building, grading, 3 failed and accepted. The return-path annotation moved below the fix box, where it stops crossing the arrow it describes, and that arrow now reaches the grade box.

…ured visual The rounds column and the cost column used to appear whole once a rail finished, which showed the answer and skipped the part that matters. Both now run off the same clock as the rail that draws them: the spend counts up continuously, a tick on the rail lights as each fix round lands, and the round counter moves with it. The cents are floored so a total cannot reach its final figure before the last round does. The two arrive together, and the rail's finish line turns green when the run touches it.

… two visuals Node labels on the loop and harness diagrams ran past the right edge of the boxes that held them. The loop nodes are now one uniform width sized to the longest label, with the arrows and the decision re-laid around them, and every label has at least 22px of clearance. The harness sub-labels are trimmed to fit theirs. The loop's return-path note moved beside the arrow it describes. On the harness diagram the feedback edge ran at y=110, straight through the preflight gate box and its text. It now takes the clear lane below that row and arrives at the agent adapter from underneath. The guarantee comparison was two independent columns, so nothing lined up across them and the two sides could not be read against each other. It is one grid now: same caption row, same box row, same outcome row. One side writes a single call, the other assembles four pieces, and when a piece goes missing the outcome on that side flips to two orders against one unit while the other side holds. Proving the grader now states what happens rather than the principle behind it: three planted inputs on the left, the verdict each one has to produce on the right, checked off in sequence. The cart-isolation step said a shopper adds a monitor, which reads two ways. It is an item now. The L1 to L5 level line is gone.

behavioural, catalogue and serialises across both decks.

The headline was a riddle and the visual stopped one step short of the thing the kicker promises. It now states the mechanism: every rule the agent writes itself can go wrong. The column captions name who enforces the rule instead of restating the riddle, the box says what the agent writes to sell the last unit once, and a bill row closes the chain the slide is there to explain: one call, one round, $4.73 against four steps, three rounds, $11.28. The plain document takes the same headline and the same causal order, and its grader section matches the deck's three planted inputs.

The deck called the target a storefront, which is half of it. The L1 spec covers accounts, a live storefront, buying, a cart, reviews and an admin area over warehouses and stock, and L2 adds fulfilment, stock movement between warehouses, returns and price changes. The hero and the premise slide now name that scope, and 'the store' works as shorthand once it is stated. Breaking it on purpose gains a seventh case, on the half that was missing: stock moves from one warehouse to another while a customer buys, and the total across warehouses has to still add up with neither warehouse going under.

The slide listed what the agent has to type, which is developer-internal and never showed why any of it matters. It shows the race now, in the same actor and step format the attack panels use, with a tab for each way it can be settled. Settled by the database: two shoppers press buy in the same millisecond, the database takes them one at a time, alice orders and bob is sold out. Settled by the agent: bob reads one in stock before alice writes, both write zero, and two orders exist against one unit. That second read is the defect, and it is on the screen rather than implied. Each tab ends on its own bill line, so the slide arrives where the kicker says it will: one call, one round, $4.73 against four steps, three rounds, $11.28.

The slide tried to teach why concurrent writes go wrong, which needs more setup than a slide carries, and it restated a point the measured slide and the scoreboard already make with numbers. Three passes at the visual left it confusing, so it is gone from both documents along with its picker and styles. The deck runs eight slides now: what is measured, the loop, the harness, what gets broken, proving the grader, the receipts, the scoreboard.

…entence The copy kept opening on a noun phrase or a qualifier and burying whoever was doing the thing. The break-it slide was the worst of it: 'Every way a store like this goes wrong' names no actor and no action, so it now says the harness breaks the store seven ways. Proving the grader reads in the order it happens: we plant a defect, and the grader has to catch it. The rest is the same pass. The hero starts with the agent building and then says what Stack Bench does to it. The premise slide is three statements instead of one sentence carrying a colon and a semicolon. The harness paragraph drops 'travel back up the dashed edge', which only means something to someone already reading the diagram. Two fragments became sentences.

'The first attempt is only the first round' was a turn of phrase where a statement belonged. It runs until it passes, same wording the plain document already used.

Proving the grader was three rows of text with an arrow in the middle, which put the claim on screen and left the evidence to the reader. The whole suite is on screen now, one tile per criterion. It sits green, a planted defect turns exactly one tile red while the other fifty hold, and an empty build blanks the board. A clean kill is something you can see in one glance rather than a phrase to accept. Receipts described the run directory in a monospace paragraph with half its words bolded white, which read as noise. It shows the directory instead: the prompt, the cost, the bug report, the videos, the hash-bound source, and the verdict file opened where it sits in the listing. The dead artifact-tree styles went with them.

The directory listing put a caption beside every filename, which is describing the evidence rather than producing it. The slide walks the chain instead: the published 51 / 51, the typed record behind it, and the recording behind that, with the scrubber running to the frame the grader marked. Each step lights as the trace reaches it. The headline follows the visual: every number traces back to a recording. The rest of the directory sits in one line underneath, named and not explained.

bradleyshep added 30 commits August 17, 2026 21:18

feat(stack-bench): allow single-sweep qualification

df48cb7

fix(stack-bench): honor single-sweep qualification

4809e11

feat(stack-bench): qualify comprehensive ecommerce L2

5194b16

chore(stack-bench): qualify neutral campaign guidance

aa41a02

fix(stack-bench): preserve qualified Spacetime fixture bytes

e80f11c

fix(stack-bench): keep campaign grading comparable

baa8dad

docs(stack-bench): use american spellings

c5a039d

behavioural, catalogue and serialises across both decks.

feat(stack-bench): run campaign attempts in parallel

146033e

feat(stack-bench): record campaign parallel scope

7e1bdcf

docs(stack-bench): name the loop slide after what the loop does

d827f9a

'The first attempt is only the first round' was a turn of phrase where a statement belonged. It runs until it passes, same wording the plain document already used.

bradleyshep added 30 commits August 25, 2026 15:52

Fix Docker clock preflight measurement

2429dd7

Pass dependency campaign plans by file

4e2cac8

Resolve dependency recipes during startup

4375dc4

Bind dependency scope in run preflight

e3b9e26

Route sandbox probes by agent capability

c7bfb81

Poll for startup data after reset

e43e023

Bound reset readiness polling

9ffd83b

Add deterministic grading shard plans

26516dc

Audit completed progression reference campaigns

37ece68

Keep campaign status output compact

55e1973

Document compact campaign status

f68435f

Align progression catalog pagination

a4b27fe

Preserve generated dependencies across reference upgrades

cbc9d36

Surface campaign process failures

74a8554

Keep stored campaign evidence inspectable

0ef3b91

Keep reference runtime config out of source

fd6aee4

Audit progression reference action inputs

46b8402

Add PostgreSQL progression action inputs

c89470f

Fix Spacetime cart action metadata

3f86319

Bind MongoDB progression action inputs

0f77128

Bind progression reference action inputs

8229f35

Bind direct grading writes to the run lease

6341931

Scale grading timeout with selected sources

5ac730c

Isolate progression grading scenarios

4f6a63a

Harden model-free dependency repair runs

1cac30c

Fix progression grading false failures

fc9f46b

Validate scenario contracts by run mode

1c5ff30

Fix PostgreSQL progression reference setup

2321cd0

Preserve Spacetime reference source identity

5509ef9

Fix progression L2 grading evidence

3cd5ecc

Navigation Menu

Uh oh!

Bradley/stackbench test - #5766

Bradley/stackbench test#5766
bradleyshep wants to merge 576 commits into
masterfrom
bradley/stackbench-test

bradleyshep commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

bradleyshep commented Aug 19, 2026

Description of Changes

Benchmark definitions and grading

Ecommerce benchmark

Runs, repairs, and campaigns

CLI, dashboard, and appliance

SpacetimeDB TypeScript SDK

SpacetimeDB CLI

SpacetimeDB skills

API and ABI breaking changes

Expected complexity level and risk

Testing

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants