Skip to content

Stop routing production to a dead Temporal engine, and label container-tool launch failures - #206

Merged
imaustink merged 2 commits into
mainfrom
fix/label-container-tool-launch-errors
Aug 14, 2026
Merged

Stop routing production to a dead Temporal engine, and label container-tool launch failures#206
imaustink merged 2 commits into
mainfrom
fix/label-container-tool-launch-errors

Conversation

@imaustink

@imaustink imaustink commented Aug 14, 2026

Copy link
Copy Markdown
Owner

Summary

The actual root cause of the recipe skill's ❌ fetch failed (found by checking the live cluster, not by guessing):

  • agent-orchestrator's logs show agent engine: temporal (http://agent-controller-temporal-engine-gateway:8080) — every turn is forwarded over HTTP to a second service.
  • kubectl describe pod on agent-controller-temporal-engine-gateway/-worker: both have sat in ImagePullBackOff for 6+ days. Their images (temporal-engine-gateway:latest, temporal-engine-worker:latest) have no registry prefix, so containerd resolves them against Docker Hub, which has never had them.
  • This production environment has no Temporal cluster at all, and was never meant to route here — but charts/agent-controller/values.yaml (the umbrella chart's own default, per docs/adr/0036's rollout) sets agent-orchestrator.config.agentEngine: temporal and temporal-engine.enabled: true, and values-production.yaml never overrode either back off. Helm deep-merges the config map key by key, so simply not mentioning agentEngine in the production values silently inherited "temporal" from the base chart.
  • Every turn was therefore forwarded to a gateway that has never once come up. The resulting dead-connection error is Node's native fetch() throwing with zero wrapping (TypeError: fetch failed), which propagated to chat verbatim — nothing recipe- or YouTube-specific about it.
  • Fix: values-production.yaml now explicitly clears agentEngine/temporalEngineUrl and sets temporal-engine.enabled: false, restoring every turn to the in-process LangGraph loop this environment actually runs on. Verified with helm template against both values files — AGENT_ENGINE/AGENT_TEMPORAL_ENGINE_URL no longer render on the orchestrator Deployment, and no temporal-engine Deployment/Service/RBAC renders at all. Diffed the full rendered manifest before/after to confirm nothing else moved.

A real, separate bug also fixed here (found while investigating, kept because it's independently correct): apps/agent-orchestrator/src/agent/graph.ts's container-Tool (ToolRun/Job) branch of runTool had no catch around containerToolLauncher.launch()/its awaited result — unlike the agent-backed branch right above it. A failure launching a ToolRun (e.g. the k8s API call itself failing) would propagate uncaught out of the graph and hit server.ts's generic finish(\❌ ${err.message}`), losing all context about which tool or that launch (not the tool) failed. Now wrapped and labeled as tool failed to launch: `, with a regression test.

What I got wrong initially

My first pass diagnosed this as the k8s-launch gap above. That doesn't hold up: @kubernetes/client-node's HTTP layer (node-fetch) formats connection errors as request to <url> failed, reason: <cause>, not the bare string the user actually saw. Checked the live cluster directly this time instead of reasoning from code alone.

Test plan

  • npx vitest run src/agent/graph.test.ts (138 passed, incl. new regression test for the launch-failure labeling)
  • npx vitest run in apps/agent-orchestrator — 3 pre-existing failures in nats-agent-channel*.test.ts, confirmed unrelated (reproduce identically on main)
  • npx tsc --noEmit — no new errors introduced
  • helm template agent-controller charts/agent-controller -f values-production.yaml — renders cleanly, diffed full output before/after the values change
  • helm upgrade against the live cluster — not run from here; flagging for confirmation before I (or anyone) applies this to production

🤖 Generated with Claude Code

https://claude.ai/code/session_01B5MkAovb1gWz4G34wSXaVC

imaustink and others added 2 commits August 13, 2026 17:28
…ncaught

The container-Tool (ToolRun/Job) branch of runTool awaited launch() and the
job result with no catch, unlike the agent-backed branch right above it. A
failure before the tool ever ran -- e.g. the k8s API call in
ToolRunLauncher.launch() failing -- propagated uncaught past every handler in
the graph and reached the SSE layer as server.ts's generic
`❌ ${err.message}`, with no indication of which tool or that launch (not the
tool itself) was what failed. This is what a user saw as a bare "❌ fetch
failed" trying to extract a recipe: recipe-scraper never got a chance to run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MkAovb1gWz4G34wSXaVC
…loyed

charts/agent-controller/values.yaml defaults agent-orchestrator.config.agentEngine
to "temporal" and temporal-engine.enabled to true (docs/adr/0036's rollout),
and values-production.yaml never overrode either back off. This production
deployment has no Temporal cluster at all, and the temporal-engine subchart's
gateway/worker images have no registry prefix (containerd resolves them
against Docker Hub, which has never had them) -- confirmed live via
`kubectl describe pod`: both Deployments have sat in ImagePullBackOff for 6+
days. Every turn was still being forwarded to that nonexistent gateway over
HTTP, and the resulting dead-connection error is what silently arrives at
end users as a bare "fetch failed" -- this is the actual cause of the "recipe
skill broke" report, not the container-tool-launch gap fixed in the prior
commit (that one's still a real bug, just not this incident).

Overrides agentEngine/temporalEngineUrl back to "" and disables the
temporal-engine subchart, restoring every turn to the in-process LangGraph
loop this environment has always actually run on. Verified with
`helm template` against both values files: AGENT_ENGINE/AGENT_TEMPORAL_ENGINE_URL
no longer render on the orchestrator Deployment and no temporal-engine
Deployment/Service/RBAC objects render at all.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MkAovb1gWz4G34wSXaVC
@imaustink imaustink changed the title Label container-tool launch failures instead of letting them escape uncaught Stop routing production to a dead Temporal engine, and label container-tool launch failures Aug 14, 2026
@imaustink
imaustink merged commit b5728f7 into main Aug 14, 2026
6 checks passed
@imaustink
imaustink deleted the fix/label-container-tool-launch-errors branch August 14, 2026 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant