feat(evals): stack lifecycle eval with Docker-less CLI arms (CLI-2398) - #281
Open
Coly010 wants to merge 2 commits into
Open
Conversation
…periments Three regression evals for CLI-2398 share one prompt (init a project, start the local stack, add a seeded `notes` table) and one scorer, differing only in the sandbox's Docker state declared beside PROMPT.md in sandbox-environment.json: 002 is the Docker-available control, 003 has the docker client but no usable daemon, 004 has no docker binary at all. The scorer checks stack readiness, the seeded rows, zero container-runtime detours (install/start/escalation attempts), records cliVersion, resolvedRuntime, timeToReadyMs and raw-socket probes as metrics, and asks an LLM judge whether the agent named the real blocker when it failed. The two Docker-less arms fail today by design; they track the gap the CLI's native managed stack is meant to close. Two experiments clone codex-gpt-5.6-luna but install the latest stable or latest beta Supabase CLI (resolved lazily from npm dist-tags) via an experiment-land LocalStackRuntime built from the sandbox package's exports, so the pinned 2.67.1 baseline, stable and beta can be compared nightly across every interface: cli regression eval. The Docker-less arms are staged with DOCKER_HOST pointed at an unbound port plus root-owned PATH shims (the CI sandbox makes the socket world-writable, so permissions alone cannot block it); for 003 the docker shim still answers --version so the managed stack's runtime probe selects Docker as it would on a real host. The existing Luna experiments skip evals that need a Docker-less sandbox. Unit tests for the pure helpers run with: pnpm --filter @supabase-evals/framework exec vitest run --root ../.. experiments/_lib evals/build-database-002-stack-lifecycle Refs: https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle
…-2398) An eval told in advance which environment it runs in, and graded toward that environment's expected story, measures conformance to an answer key rather than adaptation — and the marker-file design forced skipEval patches onto two unrelated pinned experiments just to suppress meaningless nightly fails. Collapse to one eval, build-database-002-stack-lifecycle, whose scorer asserts only environment-agnostic criteria (stack ready however it got there, seeded rows verified, zero container-runtime detours, a truthful final report) and reports resolvedRuntime as an observed metric. The forced environments become experiment variants: -cli-nodaemon (docker client present, daemon unreachable) and -cli-absent (no docker at all) wrap dockerAwareLocalStackRuntime with a `docker` option instead of reading an eval-side sandbox-environment.json, on the beta channel where the managed stack's Docker-less path lives, scoped to this scenario. The existing Luna experiments return to their upstream content and run the eval under their stock sandbox like any other. Also mirrors the upstream promptAddendum/buildSkillsPrompt API change (cd5b8d0, 03467fb): CLI agents get an empty addendum via buildToolSurfaceAddendum, matching the stock runtime. Refs: https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current Behavior
Agents in sandboxes without a usable Docker daemon have been seen reaching for
docker-install/start workarounds instead of reporting the real blocker, and nothing in this repo measured that — every local-stack eval assumes Docker works and every experiment installs the pinned CLI (2.67.1), so there was also no way to see whether the CLI's beta channel improves or regresses on a scenario over time.Expected Behavior
One environment-agnostic eval, run under experiments that force different environments and CLI channels — with no framework changes (only
evals/andexperiments/are touched; both existing Luna experiments are byte-identical tomain).Eval —
evals/build-database-002-stack-lifecycle(suite: regression,interface: cli,projectRunning: false): init a project, start the local stack, add a seedednotestable via a migration, show the rows. The scorer asserts only what must hold regardless of runtime: project initialised, migration createsnotes, stack reaches ready (however it got there), ≥ 2 rows, zero container-runtime detours (apt-get install docker,systemctl start docker,dockerd,podman,sudo, socketchmod/chown, raw Docker API writes…), and an LLM-judged truthful final report.resolvedRuntime(docker|native|none),timeToReadyMs,cliVersion,cliDetours,clearedDockerHostandrawDockerSocketProbesare reported in a metrics check, never asserted.Experiments (all Codex GPT-5.6 Luna,
suite: regression) wrap a new experiment-landdockerAwareLocalStackRuntime({ channel, docker })built entirely from@supabase-evals/sandboxexports:codex-gpt-5.6-luna/-no-skills(unchanged)2.67.1-cli-stablelatestdist-tag)interface: clinon-hosted regression eval-cli-betabetadist-tag, cut per merge todevelop)-cli-nodaemon-cli-absentdockerbinaryDocker-less arms are staged with
DOCKER_HOST=tcp://127.0.0.1:1plus root-owned PATH shims (the CI sandboxchmod 666s the socket, so permissions alone can't block it); the no-daemondockershim still answers--versionso the managed stack's runtime probe (supabase/cli#6563) selects Docker as it would on a real host. Versions resolve lazily insidestartSession(never on--dry/list), can be pinned viaSUPABASE_CLI_STABLE_VERSION/SUPABASE_CLI_BETA_VERSION, and fail loudly rather than falling back to the pin.First local runs (1 each):
resolvedRuntimecliDetours-cli-stable(2.117.0)docker-cli-nodaemon(2.118.0-beta.25)none-cli-absent(2.118.0-beta.25)noneThe two Docker-less arms fail by design today: they track the gap the CLI's native managed stack is meant to close (it's opt-in in beta). Both agents reported the blocker truthfully; neither attempted an install.
The second commit is the pivot from an earlier three-eval + marker-file shape (see the rewritten Linear issue for why); squash on merge.
Related Issue(s)
Refs https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle
Test plan
pnpm checkandpnpm format:checkgreen (smoke suite included, with a realOPENAI_API_KEY)pnpm --filter @supabase-evals/framework exec vitest run --root ../.. experiments/_lib evals/build-database-002-stack-lifecycle→ 69 passing (not part ofpnpm check;experiments/isn't covered by any package script)pnpm eval:dry -- --suite regression --experiment-suite regression: 6 experiments × 17 evals;-cli-nodaemon/-cli-absentplan exactly this eval; the pinned Luna experiments plan it with no exclusions-cli-stable,-cli-nodaemon,-cli-absent(table above); sandbox teardown leaves no containers/volumes/networksworkflow_dispatcheval-refresh withexperiments=codex-gpt-5.6-luna-cli-nodaemon,eval=build-database-002-stack-lifecycle,runs=1— confirms theDOCKER_HOSTlayering holds under the Vercel sandbox's world-writable socket (resolvedRuntimemust benone)run-evals-changedlabel) before merge