Python: [Bug]: BedrockChatClient stream=True returns the whole response in one update, so GPT-6 Sol/Luna/Astra first text arrives 3-7x later

[kimnamu]

Description

Thanks for the Bedrock connector and for keeping stream=True working end to end, including structured output. As far as I can tell, stream=True still calls the non-streaming Converse API and yields one update after the whole response has arrived, so the first text shows up only when generation is done. With the new OpenAI GPT-6 Sol/Luna/Astra models the first text arrives 3-7x later than with ConverseStream.

us-east-1, "Write about 250 words explaining how a hash map works.", n=5, median	TTFT	Total	Updates
us.openai.gpt-6-sol, raw boto3 converse_stream	1.03 s	3.32 s	-
us.openai.gpt-6-sol, BedrockChatClient stream=True (main @ f42ff01)	❌ 3.56 s	3.56 s	1
us.openai.gpt-6-luna, raw boto3 converse_stream	0.96 s	2.84 s	-
us.openai.gpt-6-luna, BedrockChatClient stream=True (main @ f42ff01)	❌ 2.83 s	2.83 s	1
us.openai.gpt-6-astra, raw boto3 converse_stream	1.88 s	9.05 s	-
us.openai.gpt-6-astra, BedrockChatClient stream=True (main @ f42ff01)	❌ 8.35 s	8.35 s	1
global.openai.gpt-6-astra, raw boto3 converse_stream	1.26 s	8.38 s	-
global.openai.gpt-6-astra, BedrockChatClient stream=True (main @ f42ff01)	❌ 9.24 s	9.24 s	1

Raw and library calls ran alternately in the same process.

Where: _inner_get_response in python/packages/bedrock/agent_framework_bedrock/_chat_client.py (L371-391): "Streaming mode - simulate streaming by yielding a single update", then asyncio.to_thread(self._invoke_converse, request), which calls converse. The package has no converse_stream call.

Impact: chat UIs and agents that stream to users wait the full generation time before showing anything. Total time is the same, so it only shows up as latency. It affects every Bedrock model.

Suggested direction: call converse_stream through the same _invoke_converse error handling and map each event to a ChatResponseUpdate: text deltas, toolUse start and input deltas (merged by call_id, like the Anthropic client streams tool input), messageStop finish reason and metadata usage. The non-streaming path stays as it is. I have a branch with tests (about 45 lines of source) and will open a PR for it; happy to adjust the shape.

Code Sample

import asyncio, time
from agent_framework.amazon import BedrockChatClient

async def main():
    client = BedrockChatClient(model="us.openai.gpt-6-sol", region="us-east-1")
    t0, arrivals = time.perf_counter(), []
    async for update in client.get_response("Count from 1 to 30 separated by spaces.", stream=True):
        if update.text:
            arrivals.append(round(time.perf_counter() - t0, 2))
    print(len(arrivals), arrivals)

asyncio.run(main())

Error Messages / Stack Traces

# sample above, main @ f42ff01: one update with the full text
1 [1.42]
# model="us.openai.gpt-6-luna"
1 [1.28]
# model="us.openai.gpt-6-astra"
1 [2.07]

No error; the delay is the only symptom.

Package Versions

agent-framework-core: 1.19.0, agent-framework-bedrock: 1.0.0b260918 (main @ f42ff01), boto3: 1.43.100

Python Version

Python 3.12

Additional Context

Raw runs (TTFT s, total s, output tokens, updates)

us.openai.gpt-6-sol | raw ConverseStream | 5 | 1.03 | 3.32 | 317 | -
us.openai.gpt-6-sol | agent-framework stream=True | 5 | 3.56 | 3.56 | 320 | [1, 1, 1, 1, 1]
   raw runs: [(1.54, 3.98, 320), (0.75, 2.86, 316), (1.03, 3.32, 309), (0.94, 3.01, 317), (1.59, 4.21, 353)]
   lib runs: [(4.04, 4.04, 320, 1), (3.56, 3.56, 343, 1), (3.17, 3.18, 317, 1), (3.79, 3.79, 342, 1), (2.94, 2.94, 293, 1)]
us.openai.gpt-6-luna | raw ConverseStream | 5 | 0.96 | 2.84 | 387 | -
us.openai.gpt-6-luna | agent-framework stream=True | 5 | 2.83 | 2.83 | 409 | [1, 1, 1, 1, 1]
   raw runs: [(0.98, 2.84, 387), (2.49, 4.2, 417), (0.91, 2.62, 385), (0.96, 2.85, 406), (0.88, 2.55, 384)]
   lib runs: [(3.35, 3.35, 419, 1), (2.75, 2.75, 389, 1), (2.82, 2.82, 409, 1), (3.32, 3.32, 474, 1), (2.83, 2.83, 377, 1)]
us.openai.gpt-6-astra | raw ConverseStream | 5 | 1.88 | 9.05 | 331 | -
us.openai.gpt-6-astra | agent-framework stream=True | 5 | 8.35 | 8.35 | 326 | [1, 1, 1, 1, 1]
   raw runs: [(2.75, 9.69, 356), (1.18, 8.47, 327), (2.08, 9.29, 327), (1.88, 9.05, 348), (1.3, 9.01, 331)]
   lib runs: [(8.47, 8.47, 316, 1), (8.35, 8.35, 325, 1), (8.61, 8.61, 326, 1), (8.32, 8.32, 329, 1), (8.31, 8.31, 333, 1)]
global.openai.gpt-6-astra | raw ConverseStream | 5 | 1.26 | 8.38 | 339 | -
global.openai.gpt-6-astra | agent-framework stream=True | 5 | 9.24 | 9.24 | 341 | [1, 1, 1, 1, 1]
   raw runs: [(1.2, 8.84, 340), (1.28, 8.38, 339), (1.81, 8.8, 344), (1.26, 6.69, 320), (1.24, 8.36, 337)]
   lib runs: [(9.43, 9.43, 331, 1), (8.51, 8.51, 328, 1), (8.17, 8.17, 341, 1), (9.34, 9.34, 357, 1), (9.24, 9.24, 360, 1)]

GPT-5.6 Sol and gpt-oss-120b also get a single update.

[github-actions]

🤖 Automated triage reproduction notes (agent-authored — trust but verify)

Agent analysis

Repro: python/packages/bedrock/agent_framework_bedrock/_chat_client.py::_inner_get_response lines 371-391 invokes blocking _invoke_converse whenever get_response(..., stream=True) is used, then yields one complete update. Minimal repro: provide a stub whose converse returns full text and whose converse_stream returns two text deltas, consume the stream, and observe ['Bedrock says hi'] instead of ['Bedrock ', 'says hi'].

• Failing test: python/packages/bedrock/tests/test_bedrock_client.py::test_streaming_response_uses_converse_stream_and_yields_text_deltas
• Files examined: python/packages/bedrock/agent_framework_bedrock/_chat_client.py, python/packages/bedrock/tests/test_bedrock_client.py, python/packages/bedrock/tests/test_bedrock_structured_output.py, python/packages/bedrock/pyproject.toml
• Tests run: Bedrock client and structured-output baseline tests, python/packages/bedrock/tests/test_bedrock_client.py::test_streaming_response_uses_converse_stream_and_yields_text_deltas
• Reported version: 1.0.0b260918
• Current version: 1.0.0b260918