Context
The MCP conformance suite already exercises real client/server interactions as part of its automated scenarios. However, the actual request/response flows are currently embedded within the suite and its library packages, which makes them difficult to inspect, reproduce, or run manually outside the automated harness.
During conformance testing, it would be useful to have a set of explicit, reusable real-call collections alongside the automated suite.
The goal is not to duplicate the conformance suite, but to make the underlying MCP traffic flows first-class artifacts that can be:
- inspected by humans,
- executed manually against a local MCP server,
- used for debugging and implementation validation,
- consumed by different clients/tools,
- and potentially generated or derived from the existing conformance scenarios.
I have tried a similar workflow while testing the Python SDK against a local MCP server and found that being able to replay concrete MCP calls directly was useful for understanding and debugging protocol behaviour. The current manual workflow, however, requires several steps and is difficult to reproduce consistently.
Proposed direction
Add a dedicated collection/artifact format containing representative real MCP calls, alongside the automated conformance tests.
For example:
conformance/
scenarios/
traceability/
real-calls/
initialize/
tools/
resources/
prompts/
elicitation/
...
Each collection could contain the actual MCP messages/flows required for a scenario, including request parameters and expected response characteristics where appropriate.
The important distinction is:
Automated conformance suite
↓
executes scenarios and validates assertions
Real-call collection
↓
surfaces the actual MCP traffic/flows
↓
human or external client can replay them
Possible formats / execution mechanisms
Postman/Newman-style collections are one possible implementation, but I don't think the proposal should be tied specifically to Postman.
The underlying collection should ideally be usable from multiple clients or execution environments, for example:
- Postman/Newman-style collection
- MCP shell / CLI
- a stateful MCP client
- a lightweight replay harness
- potentially other client implementations
This would allow the same real-call definitions to be reused without requiring every user to adopt a particular tool.
Relationship with the existing conformance suite
One question worth exploring is whether these collections should be:
- authored independently from the automated scenarios,
- generated from the existing scenarios, or
- represented in a common intermediate format from which both automated tests and human-readable/replayable calls can be produced.
Option 2 or 3 could help avoid divergence between what the conformance suite tests and what is exposed as a manual collection.
For example, if a conformance scenario produces:
initialize
↓
notifications/initialized
↓
tools/list
↓
tools/call
↓
response
the same flow could potentially be surfaced as a concrete sequence of MCP messages that a developer can inspect and replay.
This would also make the relationship between the existing traceability YAML → scenario → actual protocol traffic more visible.
Newman / Postman consideration
A Postman/Newman-style approach is attractive because collections are familiar and easy to execute, but Newman and its dependency chain have had security/vulnerability concerns.
Therefore, I would suggest evaluating alternatives rather than making Newman a hard dependency.
For example:
- a tool-neutral collection format,
- MCP-specific CLI/replay tooling,
- an MCP shell,
- or a collection format that can be imported by multiple clients.
The key requirement is the portable representation of real MCP calls, rather than Postman itself.
Why this could be useful
This would provide a middle ground between:
Unit/automated conformance testing
and
completely manual testing from scratch.
A developer implementing an MCP client/server could:
- start their local server,
- point a client/replay tool at it,
- select a conformance call collection,
- execute the real MCP interactions,
- inspect the actual traffic and responses,
- and use the same flows while debugging implementation issues.
It could also make the conformance suite easier to understand for contributors because the protocol interactions behind a scenario would be directly visible.
The intent is to complement the automated conformance suite, not replace it, by making the real protocol interactions themselves reusable and accessible.
cc @olaservo @panyam
Context
The MCP conformance suite already exercises real client/server interactions as part of its automated scenarios. However, the actual request/response flows are currently embedded within the suite and its library packages, which makes them difficult to inspect, reproduce, or run manually outside the automated harness.
During conformance testing, it would be useful to have a set of explicit, reusable real-call collections alongside the automated suite.
The goal is not to duplicate the conformance suite, but to make the underlying MCP traffic flows first-class artifacts that can be:
I have tried a similar workflow while testing the Python SDK against a local MCP server and found that being able to replay concrete MCP calls directly was useful for understanding and debugging protocol behaviour. The current manual workflow, however, requires several steps and is difficult to reproduce consistently.
Proposed direction
Add a dedicated collection/artifact format containing representative real MCP calls, alongside the automated conformance tests.
For example:
Each collection could contain the actual MCP messages/flows required for a scenario, including request parameters and expected response characteristics where appropriate.
The important distinction is:
Possible formats / execution mechanisms
Postman/Newman-style collections are one possible implementation, but I don't think the proposal should be tied specifically to Postman.
The underlying collection should ideally be usable from multiple clients or execution environments, for example:
This would allow the same real-call definitions to be reused without requiring every user to adopt a particular tool.
Relationship with the existing conformance suite
One question worth exploring is whether these collections should be:
Option 2 or 3 could help avoid divergence between what the conformance suite tests and what is exposed as a manual collection.
For example, if a conformance scenario produces:
the same flow could potentially be surfaced as a concrete sequence of MCP messages that a developer can inspect and replay.
This would also make the relationship between the existing traceability YAML → scenario → actual protocol traffic more visible.
Newman / Postman consideration
A Postman/Newman-style approach is attractive because collections are familiar and easy to execute, but Newman and its dependency chain have had security/vulnerability concerns.
Therefore, I would suggest evaluating alternatives rather than making Newman a hard dependency.
For example:
The key requirement is the portable representation of real MCP calls, rather than Postman itself.
Why this could be useful
This would provide a middle ground between:
Unit/automated conformance testing
and
completely manual testing from scratch.
A developer implementing an MCP client/server could:
It could also make the conformance suite easier to understand for contributors because the protocol interactions behind a scenario would be directly visible.
The intent is to complement the automated conformance suite, not replace it, by making the real protocol interactions themselves reusable and accessible.
cc @olaservo @panyam