Skip to content

LiteLlm streaming stores one thought part per reasoning deltaΒ #6895

Description

@benclarkeio

πŸ”΄ Required Information

Describe the Bug:

When LiteLlm streams, the aggregated (non-partial) LlmResponse joins the text buffer into a single part but passes the reasoning buffer through unjoined, so the response β€” and therefore the persisted session event β€” holds one types.Part(thought=True) per streamed reasoning delta.

_aggregate_streaming_thought_parts already does this joining, and its docstring says it produces clean parts "for session history and outbound requests", but it is only called from _content_to_message_param on the Anthropic path. The two stream finalizers pass thought_parts=list(reasoning_parts) instead.

Steps to Reproduce:

  1. Configure any LiteLlm model with reasoning enabled (xai/grok-4.6, anthropic/claude-*, OpenAI reasoning models, …).
  2. Call generate_content_async(..., stream=True).
  3. Inspect content.parts on the final response where partial is falsy.

Expected Behavior:

The same shape the non-streaming path produces: one thought part per thinking block, exactly as text becomes one part rather than one per delta.

Observed Behavior:

One thought part per delta. With xAI, whose deltas are per-token, a single thought arrives as ~40 parts:

{"role": "model", "parts": [
  {"text": "The", "thought": true},
  {"text": " user", "thought": true},
  {"text": " wants", "thought": true}
]}

Because these land in session history, every downstream consumer sees them: UIs render one thinking block per token, and prompts built from a transcript pay the per-part JSON overhead many times over.

Environment Details:

  • ADK Library Version: 2.7.1
  • Desktop OS: macOS 26.3.1
  • Python Version: 3.13.5

Model Information:

  • Are you using LiteLLM: Yes
  • Which model is being used: xai/grok-4.6 (reproduces on any reasoning model routed through LiteLLM)

🟑 Optional Information

Additional Context:

Signature boundaries matter for the fix: Anthropic closes each thinking block with a signature-only delta, and a signature only matches the text of its own block, so blocks must aggregate one part each rather than collapsing into one. _aggregate_streaming_thought_parts already splits on thought_signature, so calling it from both finalizers handles that case.

Minimal Reproduction Code:

async for response in LiteLlm(model="xai/grok-4.6").generate_content_async(
    llm_request, stream=True
):
    if not response.partial:
        print([p.text for p in response.content.parts if p.thought])

How often has this issue occurred?:

  • Always (100%)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions