Skip to content

fix(ai): route async OpenAI streaming through capture_streaming_event - #861

Merged
marandaneto merged 7 commits into
PostHog:mainfrom
ckarnell:fix/openai-async-streaming-parity
Aug 12, 2026
Merged

fix(ai): route async OpenAI streaming through capture_streaming_event#861
marandaneto merged 7 commits into
PostHog:mainfrom
ckarnell:fix/openai-async-streaming-parity

Conversation

@ckarnell

@ckarnell ckarnell commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

The two async OpenAI streaming paths (openai_async.py:242 and :567) build the $ai_generation property dict inline. Sync OpenAI, both Anthropic twins and both Gemini twins all delegate to posthog.ai.utils.capture_streaming_event. Async OpenAI streaming therefore drops $ai_usage, which #411 added "for backend cost calculations", along with $ai_tokens_source, $ai_instructions on the Responses path, and the AI Gateway double-billing warning.

The history suggests drift rather than a decision. #411 and #444 changed only utils.py and missed both async paths, while #499 edited openai_async.py directly and the property it added is on both twins today. $ai_tokens_source has been absent since February.

This routes both async paths through the shared helper instead of adding the missing keys to the inline dicts, since the duplication is what let them drift apart.

💚 How did you test it?

posthog/test/ai/openai/test_async_parity.py compares the properties captured by sync and async streaming and fails on anything the async side omits. On main it fails with drops ['$ai_tokens_source', '$ai_usage'], and it passes with this change. Anthropic runs as a control in the same test and its twins already agree, so the OpenAI difference is not an artifact of the harness.

posthog/test/ai/ is 420 passed, 88 skipped, with otel/ excluded for a dependency missing locally. ruff format is clean.

I have no live PostHog project and made no real OpenAI calls, so neither event has been watched landing in ingestion. mypy was not run.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

I direct a small studio and set this work going. The submission is mine. The finding came from a scanner we run across Python SDKs looking for sync and async twins that disagree, and Claude Code (Opus) wrote both the patch and the parity test.

Verification was deliberately not left to the same session that produced the patch. A second session re-ran the test against main and against the change, confirmed the failure text names the two dropped properties, and checked the ruff delta separately. The commit history behind the drift claim above was re-read at the commits, not carried over from the first summary.

One alternative was considered and dropped. Adding the four missing keys to the two inline dicts is a smaller diff, and it leaves in place the duplication that caused this.

The template asks for the DRI to be set as assignee. GitHub does not let an outside contributor assign anyone, so it is left blank and I am the DRI.

Both async OpenAI streaming paths built the $ai_generation property dict
inline, while sync OpenAI and both Anthropic and Gemini twins delegate to
posthog.ai.utils.capture_streaming_event. Async streaming therefore
dropped $ai_usage, $ai_tokens_source, $ai_instructions on the Responses
path, and the AI Gateway double-billing warning.

Routing through the shared helper rather than adding the keys inline,
since the duplication is what allowed the drift.

Adds a parity test that compares sync and async captured properties and
fails on main.
@ckarnell
ckarnell requested a review from a team as a code owner August 9, 2026 20:18

@marandaneto marandaneto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline review finding.

Comment thread posthog/test/ai/openai/test_async_parity.py Outdated
@marandaneto
marandaneto requested review from a team August 10, 2026 08:10
@marandaneto

Copy link
Copy Markdown
Member

moving to draft until comments are resolved

@marandaneto
marandaneto marked this pull request as draft August 10, 2026 08:11
@ckarnell

Copy link
Copy Markdown
Contributor Author

Done. You were right to push back on the suppression, since it would have left the same import doing the same thing with the warning turned off.

Both fixtures now live in posthog/test/ai/openai/conftest.py, so pytest supplies them by discovery and no test module imports or rebinds the names. The noqa comments are gone. mock_client and streaming_tool_call_chunks moved out of test_openai.py unchanged, along with the two chunk-type imports only they used.

ruff check passes on those files. posthog/test/ai is 420 passed and 88 skipped, unchanged.

The API snapshot was also out of date and is regenerated. Routing the async streaming path through capture_streaming_event dropped the get_model_params alias from openai_async, which is the one line that changed.

@ckarnell
ckarnell marked this pull request as ready for review August 10, 2026 08:16
ckarnell and others added 3 commits August 10, 2026 10:29
The test module imports mock_client and streaming_tool_call_chunks from
test_openai.py and then takes them as parameters, which ruff reads as
redefinition. No other test file in the repo imports fixtures across
modules, so rather than move them into a conftest and restructure the test
layout, the three call sites carry a targeted noqa.

The public API snapshot dropped the get_model_params alias from
openai_async, which this PR removed by routing streaming through
capture_streaming_event. Regenerated with the repo's own script.
Replaces the noqa suppressions from the previous commit, per review. The two
fixtures now live in conftest.py so pytest supplies them by discovery, which
means no test module imports or rebinds the names and Ruff F811 has nothing
to fire on.

mock_client and streaming_tool_call_chunks moved out of test_openai.py
unchanged, with the imports they need. Two chunk-type imports that only the
moved fixture used are dropped from test_openai.py.

posthog/test/ai is 420 passed, 88 skipped, the same as before the move.
@marandaneto
marandaneto force-pushed the fix/openai-async-streaming-parity branch from c86b106 to ea6ebc0 Compare August 10, 2026 08:30

@marandaneto marandaneto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline review findings.

Comment thread posthog/ai/openai/openai_async.py
Comment thread posthog/ai/openai/openai_async.py
Comment thread posthog/ai/openai/openai_async.py
@marandaneto
marandaneto requested a review from a team August 10, 2026 10:13
@marandaneto
marandaneto marked this pull request as draft August 11, 2026 06:41
@marandaneto
marandaneto marked this pull request as ready for review August 11, 2026 11:32
@marandaneto

Copy link
Copy Markdown
Member

pushed some changes

@ckarnell

Copy link
Copy Markdown
Contributor Author

Thanks for cleaning it up, the re-export restore and the zero value assertions both look right to me.

@marandaneto
marandaneto merged commit 9c4fd84 into PostHog:main Aug 12, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants