Stop routing production to a dead Temporal engine, and label container-tool launch failures - #206
Merged
Merged
Conversation
…ncaught
The container-Tool (ToolRun/Job) branch of runTool awaited launch() and the
job result with no catch, unlike the agent-backed branch right above it. A
failure before the tool ever ran -- e.g. the k8s API call in
ToolRunLauncher.launch() failing -- propagated uncaught past every handler in
the graph and reached the SSE layer as server.ts's generic
`❌ ${err.message}`, with no indication of which tool or that launch (not the
tool itself) was what failed. This is what a user saw as a bare "❌ fetch
failed" trying to extract a recipe: recipe-scraper never got a chance to run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MkAovb1gWz4G34wSXaVC
…loyed charts/agent-controller/values.yaml defaults agent-orchestrator.config.agentEngine to "temporal" and temporal-engine.enabled to true (docs/adr/0036's rollout), and values-production.yaml never overrode either back off. This production deployment has no Temporal cluster at all, and the temporal-engine subchart's gateway/worker images have no registry prefix (containerd resolves them against Docker Hub, which has never had them) -- confirmed live via `kubectl describe pod`: both Deployments have sat in ImagePullBackOff for 6+ days. Every turn was still being forwarded to that nonexistent gateway over HTTP, and the resulting dead-connection error is what silently arrives at end users as a bare "fetch failed" -- this is the actual cause of the "recipe skill broke" report, not the container-tool-launch gap fixed in the prior commit (that one's still a real bug, just not this incident). Overrides agentEngine/temporalEngineUrl back to "" and disables the temporal-engine subchart, restoring every turn to the in-process LangGraph loop this environment has always actually run on. Verified with `helm template` against both values files: AGENT_ENGINE/AGENT_TEMPORAL_ENGINE_URL no longer render on the orchestrator Deployment and no temporal-engine Deployment/Service/RBAC objects render at all. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5MkAovb1gWz4G34wSXaVC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The actual root cause of the recipe skill's
❌ fetch failed(found by checking the live cluster, not by guessing):agent-orchestrator's logs showagent engine: temporal (http://agent-controller-temporal-engine-gateway:8080)— every turn is forwarded over HTTP to a second service.kubectl describe podonagent-controller-temporal-engine-gateway/-worker: both have sat inImagePullBackOfffor 6+ days. Their images (temporal-engine-gateway:latest,temporal-engine-worker:latest) have no registry prefix, so containerd resolves them against Docker Hub, which has never had them.charts/agent-controller/values.yaml(the umbrella chart's own default, per docs/adr/0036's rollout) setsagent-orchestrator.config.agentEngine: temporalandtemporal-engine.enabled: true, andvalues-production.yamlnever overrode either back off. Helm deep-merges theconfigmap key by key, so simply not mentioningagentEnginein the production values silently inherited "temporal" from the base chart.fetch()throwing with zero wrapping (TypeError: fetch failed), which propagated to chat verbatim — nothing recipe- or YouTube-specific about it.values-production.yamlnow explicitly clearsagentEngine/temporalEngineUrland setstemporal-engine.enabled: false, restoring every turn to the in-process LangGraph loop this environment actually runs on. Verified withhelm templateagainst both values files —AGENT_ENGINE/AGENT_TEMPORAL_ENGINE_URLno longer render on the orchestrator Deployment, and no temporal-engine Deployment/Service/RBAC renders at all. Diffed the full rendered manifest before/after to confirm nothing else moved.A real, separate bug also fixed here (found while investigating, kept because it's independently correct):
apps/agent-orchestrator/src/agent/graph.ts's container-Tool (ToolRun/Job) branch ofrunToolhad nocatcharoundcontainerToolLauncher.launch()/its awaited result — unlike the agent-backed branch right above it. A failure launching a ToolRun (e.g. the k8s API call itself failing) would propagate uncaught out of the graph and hitserver.ts's genericfinish(\❌ ${err.message}`), losing all context about which tool or that launch (not the tool) failed. Now wrapped and labeled astool failed to launch: `, with a regression test.What I got wrong initially
My first pass diagnosed this as the k8s-launch gap above. That doesn't hold up:
@kubernetes/client-node's HTTP layer (node-fetch) formats connection errors asrequest to <url> failed, reason: <cause>, not the bare string the user actually saw. Checked the live cluster directly this time instead of reasoning from code alone.Test plan
npx vitest run src/agent/graph.test.ts(138 passed, incl. new regression test for the launch-failure labeling)npx vitest runinapps/agent-orchestrator— 3 pre-existing failures innats-agent-channel*.test.ts, confirmed unrelated (reproduce identically onmain)npx tsc --noEmit— no new errors introducedhelm template agent-controller charts/agent-controller -f values-production.yaml— renders cleanly, diffed full output before/after the values changehelm upgradeagainst the live cluster — not run from here; flagging for confirmation before I (or anyone) applies this to production🤖 Generated with Claude Code
https://claude.ai/code/session_01B5MkAovb1gWz4G34wSXaVC