fix(ai): route async OpenAI streaming through capture_streaming_event - #861
Conversation
Both async OpenAI streaming paths built the $ai_generation property dict inline, while sync OpenAI and both Anthropic and Gemini twins delegate to posthog.ai.utils.capture_streaming_event. Async streaming therefore dropped $ai_usage, $ai_tokens_source, $ai_instructions on the Responses path, and the AI Gateway double-billing warning. Routing through the shared helper rather than adding the keys inline, since the duplication is what allowed the drift. Adds a parity test that compares sync and async captured properties and fails on main.
|
moving to draft until comments are resolved |
|
Done. You were right to push back on the suppression, since it would have left the same import doing the same thing with the warning turned off. Both fixtures now live in
The API snapshot was also out of date and is regenerated. Routing the async streaming path through |
The test module imports mock_client and streaming_tool_call_chunks from test_openai.py and then takes them as parameters, which ruff reads as redefinition. No other test file in the repo imports fixtures across modules, so rather than move them into a conftest and restructure the test layout, the three call sites carry a targeted noqa. The public API snapshot dropped the get_model_params alias from openai_async, which this PR removed by routing streaming through capture_streaming_event. Regenerated with the repo's own script.
Replaces the noqa suppressions from the previous commit, per review. The two fixtures now live in conftest.py so pytest supplies them by discovery, which means no test module imports or rebinds the names and Ruff F811 has nothing to fire on. mock_client and streaming_tool_call_chunks moved out of test_openai.py unchanged, with the imports they need. Two chunk-type imports that only the moved fixture used are dropped from test_openai.py. posthog/test/ai is 420 passed, 88 skipped, the same as before the move.
c86b106 to
ea6ebc0
Compare
|
pushed some changes |
|
Thanks for cleaning it up, the re-export restore and the zero value assertions both look right to me. |
💡 Motivation and Context
The two async OpenAI streaming paths (
openai_async.py:242and:567) build the$ai_generationproperty dict inline. Sync OpenAI, both Anthropic twins and both Gemini twins all delegate toposthog.ai.utils.capture_streaming_event. Async OpenAI streaming therefore drops$ai_usage, which #411 added "for backend cost calculations", along with$ai_tokens_source,$ai_instructionson the Responses path, and the AI Gateway double-billing warning.The history suggests drift rather than a decision. #411 and #444 changed only
utils.pyand missed both async paths, while #499 editedopenai_async.pydirectly and the property it added is on both twins today.$ai_tokens_sourcehas been absent since February.This routes both async paths through the shared helper instead of adding the missing keys to the inline dicts, since the duplication is what let them drift apart.
💚 How did you test it?
posthog/test/ai/openai/test_async_parity.pycompares the properties captured by sync and async streaming and fails on anything the async side omits. Onmainit fails withdrops ['$ai_tokens_source', '$ai_usage'], and it passes with this change. Anthropic runs as a control in the same test and its twins already agree, so the OpenAI difference is not an artifact of the harness.posthog/test/ai/is 420 passed, 88 skipped, withotel/excluded for a dependency missing locally.ruff formatis clean.I have no live PostHog project and made no real OpenAI calls, so neither event has been watched landing in ingestion.
mypywas not run.📝 Checklist
If releasing new changes
sampo addto generate a changeset file🤖 Agent context
Autonomy: Human-driven (agent-assisted)
I direct a small studio and set this work going. The submission is mine. The finding came from a scanner we run across Python SDKs looking for sync and async twins that disagree, and Claude Code (Opus) wrote both the patch and the parity test.
Verification was deliberately not left to the same session that produced the patch. A second session re-ran the test against
mainand against the change, confirmed the failure text names the two dropped properties, and checked the ruff delta separately. The commit history behind the drift claim above was re-read at the commits, not carried over from the first summary.One alternative was considered and dropped. Adding the four missing keys to the two inline dicts is a smaller diff, and it leaves in place the duplication that caused this.
The template asks for the DRI to be set as assignee. GitHub does not let an outside contributor assign anyone, so it is left blank and I am the DRI.