Skip to content

fix(proto): one ChatItem serializer that carries every declared field - #6756

Open
u9g wants to merge 3 commits into
mainfrom
fix/remote-session-chat-item-proto-fields
Open

fix(proto): one ChatItem serializer that carries every declared field#6756
u9g wants to merge 3 commits into
mainfrom
fix/remote-session-chat-item-proto-fields

Conversation

@u9g

@u9g u9g commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

_chat_item_to_proto in voice/remote_session.py dropped FunctionCallOutput.id. The proto declares the field, the sibling serializer in telemetry/traces.py set it, agents-js set it, and every other branch of the same function set id. The identical omission of name from this constructor was fixed once already in #6491.

Nothing reads that id today — livekit-cli renders function call outputs by name/output/is_error — so it was silent data loss rather than a visible break.

Why there was a second copy to disagree with

Rather than add the one field, I measured both serializers: saturate every ChatItem field with a non-default value, serialize, then walk the proto descriptor for whatever came back unset. On main:

remote_session
  message               transcript_confidence, extra, metrics.started_speaking_at,
                        metrics.stopped_speaking_at, created_at
  function_call         created_at
  function_call_output  id, created_at
  agent_handoff         created_at
  agent_config_update   created_at

telemetry/traces
  message               metrics.llm_node_tps, metrics.llm_node_ttfs

Each copy was missing a different subset. remote_session never got the speaking-at timestamps because its _metrics_to_proto splatted scalars only and those two are Timestamps; traces.py never got tps/ttfs because #6373 added them to the other list. Neither author was wrong — the mapping was written twice and drifted wherever the other one didn't.

The three commits

refactor: move proto serialization into its own module — both copies into livekit/agents/_proto.py, otherwise verbatim. A leaf: it imports neither llm, voice nor telemetry at module scope, so anything that puts a ChatItem on the wire can share it. The one non-mechanical part is that the voice copy now dispatches on ChatItem's discriminator rather than isinstance; importing the classes pulls in llm, llm reaches telemetry, and the import cycles.

fix(proto): carry every field the wire format declaresencode_by_name reads the fields the proto declares off the source object, so one implementation serves both paths and a field added to the schema is carried without editing a call site. _session_usage_to_proto's five variants collapse to a table over the same helper.

test(proto): fail when a declared field is dropped — the guard, below.

Behavior changes

  • Every field in the table above now reaches the wire.
  • Chat item timestamps move from FromMilliseconds to FromNanoseconds. created_at is time.time(), so sub-millisecond components were being truncated.
  • Message content is one entry per part rather than the parts joined with \n. The proto field is repeated; joining collapsed them. This is what traces.py and agents-js already sent — the RemoteSession path is the one that changes.

Tests

tests/test_chat_item_proto_coverage.py saturates every ChatItem, model usage and metrics source, encodes it, then walks the result against the proto descriptor; anything left unset is a drop. That is what makes encode_by_name safe, since a field it cannot find on the source is skipped rather than raised — verified against a deliberately stale source, which the guard reports field by field.

Fixture timestamps are sub-millisecond and message content is multi-part, so a serializer that truncates or joins fails rather than passing on a value that hides the difference.

1885 passed. The 15 failures in test_realtime/test_openai_realtime_model.py and test_realtime/test_xai_realtime_model.py are pre-existing: a pristine origin/main checkout in the same environment fails the same 15.

encode_by_name costs mypy coverage on the copied assignments, which is the deliberate trade — the guard covers the same ground at test time rather than in the editor. Where it collided with typing outright, at ModelUsage's oneof, the code uses CopyFrom rather than a dict splat, so scripts/check_types.py still passes clean.

agents-js

livekit/agents-js#2247 does the same there, and its telemetry log now matches this one field for field — verified by encoding the same five items through both SDKs and diffing the JSON.

Follow-ups, not in this PR

MetricsReport has no playback_latency, so neither SDK can carry it; agents-js was emitting it into telemetry only because its hand-written types had no link to the schema. It needs a field in livekit/protocol.

The function_tools_executed handler still builds FunctionCallOutput without id (remote_session.py:528), as does the agents-js equivalent. Consistent across SDKs, so left alone.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

u9g added 3 commits August 9, 2026 10:40
Both copies of the ChatItem mapping now sit in one file, which is the point:
they were written twice, in voice and in telemetry, and had drifted.

The move is otherwise verbatim, except that the voice copy dispatches on
ChatItem's discriminator rather than isinstance. Importing the classes would
pull llm in, and llm reaches telemetry, so the module could not stay a leaf.
The voice copy dropped FunctionCallOutput.id, created_at on every item,
transcript_confidence and extra on messages, and the speaking-at metrics;
the telemetry copy dropped llm_node_tps and llm_node_ttfs and truncated
timestamps to milliseconds. Each was missing whatever the other had.

encode_by_name reads the fields the proto declares off the source object,
so one implementation serves both paths and a field added to the schema is
carried without editing a call site. Message content follows the telemetry
copy, one entry per part, because the proto field is repeated and joining
the parts collapsed them into one.
Every ChatItem, model usage and metrics source is saturated, encoded, then
walked against the proto descriptor; anything left unset is a drop.

Timestamps in the fixture are sub-millisecond and message content is
multi-part, so a serializer that truncates or joins fails rather than
passing on a value that hides the difference.
@u9g
u9g force-pushed the fix/remote-session-chat-item-proto-fields branch from c8aa129 to 7aba481 Compare August 9, 2026 14:53
@u9g u9g changed the title fix(voice): complete the ChatItem proto serializers and guard them fix(proto): one ChatItem serializer that carries every declared field Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant