test(agent): fail hung/slow tests loudly (pytest-timeout + aws_session barrier bound)#598
Open
isadeks wants to merge 3 commits into
Open
test(agent): fail hung/slow tests loudly (pytest-timeout + aws_session barrier bound)#598isadeks wants to merge 3 commits into
isadeks wants to merge 3 commits into
Conversation
…(ROOT of the ECS flaky hang) test_concurrent_first_call_builds_once used a bare threading.Barrier(20).wait() + unbounded t.join(). Barrier(N).wait() blocks FOREVER unless all N threads arrive; under ECS container memory pressure a worker thread can be reaped (or thread creation throttled) before reaching the barrier, so every survivor hangs on wait() and the main thread hangs in join() — stalling the whole `mise run build` until the 3600s ceiling. THIS is the ECS-only flaky hang chased across ABCA-684/686/688 (pytest-timeout signal-method is the backstop; this is the root cause). Bounded the barrier wait (timeout=30 → BrokenBarrierError) and the joins (timeout=60), so a missing thread fails the test fast+loud instead of deadlocking. Stress-ran 20x locally: no hang. The test_attachments name pytest-timeout stamped earlier was a red herring — the tracker reports whichever test was last seen, not the barrier-blocked one. (cherry picked from commit 7ff91bf) (cherry picked from commit 85d9881)
Agent pytest can hang on the ECS build box (an env-specific test blocking on network/subprocess/Bedrock with no timeout of its own), stalling the pre-agent baseline until the 3600s BUILD_VERIFY_TIMEOUT_S guard — turning one flaky test into an un-diagnosable ~60-min stall (ABCA-684/686). Add pytest-timeout with a 120s per-test cap so the offending test fails LOUDLY with a dumped traceback and the suite moves on. Agent-side only: the jest --maxWorkers cap from the same source commit is Mac-local build-box tuning (core-relative worker count), deliberately NOT brought to main — CI containers have a different core/RAM profile. (cherry-picked agent hunks of 5ea9670 from linear-vercel) (cherry picked from commit bfcd280)
…ually ABORTED The thread method only PRINTS the hung test's stack at the 120s cap — it cannot interrupt a test blocked in a C-level/socket syscall. Proven live on ABCA-688: test_attachments dumped its stack at 120s, then the build kept hanging ~55 min until the platform's 3600s build-verify ceiling finally killed it. pytest runs single-threaded in the main thread here, so SIGALRM (method=signal) interrupts even a syscall-blocked test and fails it in 120s as intended. Full agent suite still 1315 passed (4.5s), plugin active. (cherry picked from commit 803c44b) (cherry picked from commit 1d20679)
isadeks
added
enhancement
scottschreckengaust
approved these changes
Jul 14, 2026
scottschreckengaust
left a comment
scottschreckengaust
left a comment
Contributor
There was a problem hiding this comment.
Review — Principal Architect (Stack A: ECS substrate, PR 4/4 · leaf)
Base verified against fresh origin PR head (diffed vs #597).
Verdict: Approve ✅
Exemplary test-hygiene work — the kind that pays for itself the first time CI would otherwise hang for an hour.
pytest-timeoutwithtimeout_method = "signal"— the comment nails the subtle, correct distinction:method="signal"(SIGALRM) actually aborts a syscall-blocked test in the single-threaded main-thread runner, whereasmethod="thread"only prints the stack and lets the build keep hanging to the 3600s ceiling (proven live on ABCA-688). Choosing signal is the real fix, not the cargo-cult default.timeout=120per test is a sane cap.- Bounded barrier/join in
test_aws_session.py—Barrier(20).wait(timeout=30)+join(timeout=60)fixes the root cause the pytest-timeout cap only symptom-treats: a bareBarrier.wait()blocks forever if one of 20 threads never arrives (worker reaped under memory pressure / thread-creation throttling), hanging the wholemise run build. A timeout raisesBrokenBarrierErrorso the test fails fast, and a leftover live thread trips theresults != 20assertion rather than blocking. Clean.
This is pure test-infrastructure (no product code), so the change is its own coverage — nothing further to add. uv.lock updated in lockstep with pyproject.toml.
Governance note
Branch pr/ecs-test-hygiene carries no issue number (ADR-003) — same item as the rest of the stack; attach an approved issue / conforming branch before merge. Code is approve-ready.
CI
All green (title, CodeQL ×3, dead-code advisory, secrets/deps/actions, build (agentcore)); MERGEABLE.
🤖 Generated with Claude Code
Contributor
Author
|
Thanks for the approve — no code changes needed here (nit-free). Governance (approved issue + conforming branch name) tracked with the rest of Stack A; will link/rename before merge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test-suite hygiene so a hung or slow agent test fails loudly and fast instead of silently burning the whole build-verify budget — surfaced when the agent pytest suite stalled >40 min on the ECS build box before the 3600s ceiling killed it.
What changed
fix(build): addpytest-timeout(120s per-test cap) — a single test blocking on an unmocked network/subprocess/Bedrock call otherwise hangsmise run builduntil the build-verify ceiling. Now the offending test fails with a dumped traceback in 120s and the suite moves on. (Agent-side only — the jest--maxWorkerstuning from the same source work is deliberately not included; that's local build-box tuning, and CI containers have a different core/RAM profile.)fix(agent): pytest-timeout methodthread→signal— so a hung test is actually aborted (the thread method can't interrupt a blocked C call).fix(agent): bound theaws_sessionconcurrency test's Barrier + joins — the root of an intermittent ECS test hang; the test could block forever if a worker never reached the barrier.Testing
mise run buildgreen; agent suite passes with the timeout plugin active.Notes for reviewers
main's jest config untouched by design.