Skip to content

fix(mcp): recover pooled session after server-side termination - #6933

Open
gioboa wants to merge 1 commit into
google:mainfrom
gioboa:fix/6822
Open

fix(mcp): recover pooled session after server-side termination#6933
gioboa wants to merge 1 commit into
google:mainfrom
gioboa:fix/6822

Conversation

@gioboa

@gioboa gioboa commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:

When a streamable-HTTP MCP server terminates a session server-side (restart or idle eviction), it answers 404 for the stored mcp-session-id. The MCP SDK surfaces this as an McpError("Session terminated") injected into the read stream, while the local read/write streams stay open and the background session task stays alive. MCPSessionManager's health checks (_is_session_disconnected and the task-aliveness probe) therefore keep approving the dead pooled session, and every later tool call on the toolset fails with {"error": "MCP tool execution failed: Session terminated"} forever. The only recovery was tearing down and re-creating the whole McpToolset from application code.

Solution:

Drop the pooled session from the pool when the server reports it terminated, and retry the tool call once on a fresh session:

  • mcp_session_manager.py: new _is_session_terminated_error() predicate (matches only the SDK's session-terminated McpError, not ordinary protocol errors) and MCPSessionManager._invalidate_session(headers), which drops and cleans the pooled entry under the session lock.
  • mcp_tool.py: the guarded call is extracted into _call_tool_on_session(); on a session-terminated error it invalidates the pool entry and re-raises, and _run_async_impl retries once on a fresh session.

The single retry cannot duplicate a remote side effect: the server rejected the request with 404 before running the tool. This is deliberately distinct from the ambiguous ConnectionError case, which remains non-retried. Ordinary McpErrors keep the pooled session (and its server-side state) and are not retried.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

New tests:

  • test_run_async_impl_recovers_from_terminated_session — dead pooled session is invalidated and the call succeeds on a fresh session.
  • test_run_async_impl_does_not_retry_other_mcp_errors — ordinary MCP protocol errors keep the session and are not retried.
  • test_run_async_impl_terminated_session_twice_raises — a second terminated-session failure surfaces instead of looping.
  • test_is_session_terminated_error — predicate matches only the SDK's session-terminated error.
  • test_invalidate_session_drops_pooled_session — pool entry is removed, its exit stack closed, and re-invalidation is a no-op.
uv run pytest tests/unittests/tools/mcp_tool/ -q
351 passed

Formatting and static checks: pyink, isort, and ruff pass; mypy reports only pre-existing findings (none on changed lines).

Manual End-to-End (E2E) Tests:

Reproduced and verified against a real stateful streamable-HTTP MCP server (FastMCP with a ping tool), driven through McpToolset / McpTool.run_async:

  1. Start the server, create an McpToolset with StreamableHTTPConnectionParams, and make one successful call (pong) so the session is pooled.
  2. SIGKILL the server and restart it on the same port — the stored mcp-session-id is now unknown to the server (same 404 path as idle-session eviction).
  3. Call the tool again.

Before this change, step 3 and every later call returned {'error': 'MCP tool execution failed: Session terminated'} indefinitely, even with the server healthy. With this change, the first post-restart call already succeeds:

call 1 (fresh server): {'content': [{'type': 'text', 'text': 'pong'}], ..., 'isError': False}
server restarted; old session id now unknown to the server
call 2 (post-restart): {'content': [{'type': 'text', 'text': 'pong'}], ..., 'isError': False}
call 3 (post-restart): {'content': [{'type': 'text', 'text': 'pong'}], ..., 'isError': False}

Negative control: with the new predicate forced to False, the identical E2E run shows the original never-recovers behavior, confirming the fix is what restores recovery.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants