fix(mcp): recover pooled session after server-side termination - #6933
Open
gioboa wants to merge 1 commit into
Open
fix(mcp): recover pooled session after server-side termination#6933gioboa wants to merge 1 commit into
gioboa wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
_is_session_disconnectedonly checks local stream flags (scope of closed #3321, still broken in v2.7.1) #6822Problem:
When a streamable-HTTP MCP server terminates a session server-side (restart or idle eviction), it answers 404 for the stored
mcp-session-id. The MCP SDK surfaces this as anMcpError("Session terminated")injected into the read stream, while the local read/write streams stay open and the background session task stays alive.MCPSessionManager's health checks (_is_session_disconnectedand the task-aliveness probe) therefore keep approving the dead pooled session, and every later tool call on the toolset fails with{"error": "MCP tool execution failed: Session terminated"}forever. The only recovery was tearing down and re-creating the wholeMcpToolsetfrom application code.Solution:
Drop the pooled session from the pool when the server reports it terminated, and retry the tool call once on a fresh session:
mcp_session_manager.py: new_is_session_terminated_error()predicate (matches only the SDK's session-terminatedMcpError, not ordinary protocol errors) andMCPSessionManager._invalidate_session(headers), which drops and cleans the pooled entry under the session lock.mcp_tool.py: the guarded call is extracted into_call_tool_on_session(); on a session-terminated error it invalidates the pool entry and re-raises, and_run_async_implretries once on a fresh session.The single retry cannot duplicate a remote side effect: the server rejected the request with 404 before running the tool. This is deliberately distinct from the ambiguous
ConnectionErrorcase, which remains non-retried. OrdinaryMcpErrors keep the pooled session (and its server-side state) and are not retried.Testing Plan
Unit Tests:
New tests:
test_run_async_impl_recovers_from_terminated_session— dead pooled session is invalidated and the call succeeds on a fresh session.test_run_async_impl_does_not_retry_other_mcp_errors— ordinary MCP protocol errors keep the session and are not retried.test_run_async_impl_terminated_session_twice_raises— a second terminated-session failure surfaces instead of looping.test_is_session_terminated_error— predicate matches only the SDK's session-terminated error.test_invalidate_session_drops_pooled_session— pool entry is removed, its exit stack closed, and re-invalidation is a no-op.Formatting and static checks: pyink, isort, and ruff pass; mypy reports only pre-existing findings (none on changed lines).
Manual End-to-End (E2E) Tests:
Reproduced and verified against a real stateful streamable-HTTP MCP server (FastMCP with a
pingtool), driven throughMcpToolset/McpTool.run_async:McpToolsetwithStreamableHTTPConnectionParams, and make one successful call (pong) so the session is pooled.SIGKILLthe server and restart it on the same port — the storedmcp-session-idis now unknown to the server (same 404 path as idle-session eviction).Before this change, step 3 and every later call returned
{'error': 'MCP tool execution failed: Session terminated'}indefinitely, even with the server healthy. With this change, the first post-restart call already succeeds:Negative control: with the new predicate forced to
False, the identical E2E run shows the original never-recovers behavior, confirming the fix is what restores recovery.Checklist
Additional context
McpError/"Session terminated" handling discussed there never landed;_is_session_disconnectedcannot detect this case from local state, which is why the invalidation happens at the call site where the error is observed.