test: strengthen SDK regression coverage and fix MCP client lifecycle - #283
marandaneto wants to merge 1 commit into
Conversation
Test coverage before and afterCompared baseline
Test results
There are no RSpec failures. The two optional integrations were silently omitted from the default baseline. They now appear as pending without their dependencies and pass with the OpenTelemetry bundle. The increase also includes three existing JSON compatibility cases that now run in separate processes during normal discovery. The main improvement is stronger assertions, not just more executed lines. Six controlled defects passed the old tests and failed the repaired tests: returning nil for remote config, sending a partial batch too early, failing a child-process flush, skipping middleware insertion, overwriting an application-owned session header, and not restarting the OpenTelemetry background worker. The three new MCP lifecycle cases also failed before the production fix and passed afterward. Measurement limitsThese figures cover 62 loaded production files in the parent process under Version files loaded before coverage starts, the Rails generator and template, and the Rails entry wrapper are outside this measurement. These are not whole-repository coverage percentages. The supported-Ruby CI matrix still needs to run. |
posthog-ruby-sync Compliance ReportDate: 2026-09-26T15:28:11.073371+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 10ms |
| Format Validation.Event Has Uuid | ✅ | 7ms |
| Format Validation.Event Has Lib Properties | ✅ | 7ms |
| Format Validation.Distinct Id Is String | ✅ | 9ms |
| Format Validation.Token Is Present | ✅ | 8ms |
| Format Validation.Custom Properties Preserved | ✅ | 7ms |
| Format Validation.Event Has Timestamp | ✅ | 9ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 8ms |
| Retry Behavior.Retries On 503 | ✅ | 5317ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2008ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2011ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8016ms |
| Retry Behavior.Implements Backoff | ✅ | 15310ms |
| Retry Behavior.Retries On 500 | ✅ | 5164ms |
| Retry Behavior.Retries On 502 | ✅ | 5156ms |
| Retry Behavior.Retries On 504 | ✅ | 5117ms |
| Retry Behavior.Max Retries Respected | ✅ | 15515ms |
| Deduplication.Generates Unique Uuids | ✅ | 22ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5110ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10257ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5146ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 19ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 9ms |
| Compression.Sends Gzip When Enabled | ✅ | 7ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 6ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 4ms |
| Batch Format.Multiple Events Batched Together | ❌ | 17ms |
| Error Handling.Does Not Retry On 403 | ✅ | 2006ms |
| Error Handling.Does Not Retry On 413 | ✅ | 2010ms |
| Error Handling.Retries On 408 | ✅ | 5150ms |
Failures
batch_format.multiple_events_batched_together
Expected 1 requests, got 5
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 9ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 9ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 10ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 7ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 7ms |
| Request Payload.Groups Round Trip | ✅ | 6ms |
| Request Payload.Groups Default To Empty Object | ✅ | 7ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 6ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 7ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 7ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 2ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 6ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 10ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 7ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 110ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 144ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 9ms |
Failures
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']
posthog-ruby-async Compliance ReportDate: 2026-09-26T15:28:22.492717+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 107ms |
| Format Validation.Event Has Uuid | ✅ | 106ms |
| Format Validation.Event Has Lib Properties | ✅ | 108ms |
| Format Validation.Distinct Id Is String | ✅ | 106ms |
| Format Validation.Token Is Present | ✅ | 107ms |
| Format Validation.Custom Properties Preserved | ✅ | 106ms |
| Format Validation.Event Has Timestamp | ✅ | 106ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 8ms |
| Retry Behavior.Retries On 503 | ✅ | 5406ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2110ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2108ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8016ms |
| Retry Behavior.Implements Backoff | ✅ | 15721ms |
| Retry Behavior.Retries On 500 | ✅ | 5211ms |
| Retry Behavior.Retries On 502 | ✅ | 5207ms |
| Retry Behavior.Retries On 504 | ✅ | 5211ms |
| Retry Behavior.Max Retries Respected | ✅ | 15621ms |
| Deduplication.Generates Unique Uuids | ✅ | 110ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5213ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10316ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5213ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 109ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 107ms |
| Compression.Sends Gzip When Enabled | ✅ | 105ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 105ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 4ms |
| Batch Format.Multiple Events Batched Together | ✅ | 109ms |
| Error Handling.Does Not Retry On 403 | ✅ | 2108ms |
| Error Handling.Does Not Retry On 413 | ✅ | 2108ms |
| Error Handling.Retries On 408 | ✅ | 5211ms |
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 106ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 107ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 106ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 107ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 108ms |
| Request Payload.Groups Round Trip | ✅ | 107ms |
| Request Payload.Groups Default To Empty Object | ✅ | 107ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 107ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 106ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 106ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 3ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 105ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 109ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 106ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 253ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 208ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 108ms |
Failures
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']
|
[Medium risk] Fixes client lifecycle tracking for subclasses and strengthens test coverage. The PR appears safe to merge, though the instance-count guard should be corrected to preserve duplicate-client warnings. Reviews (1) · Last reviewed commit: "test: strengthen SDK regression coverage..." |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| flushed = false | ||
| begin | ||
| self.class._decrement_instance_count(@api_key) unless @disabled | ||
| PostHog::Client._decrement_instance_count(@api_key) unless @disabled |
There was a problem hiding this comment.
Clients created with test_mode: true or disable_singleton_warning: true do not increment the shared count, but shutdown still decrements it. If another client with the same API key remains active, a later client can miss the duplicate-client warning. Decrement only for clients that were counted.
Knowledge Base Used: Client lifecycle and public API
Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/posthog/client.rb
Line: 950
Comment:
**Uncounted clients reduce tracking**
Clients created with `test_mode: true` or `disable_singleton_warning: true` do not increment the shared count, but shutdown still decrements it. If another client with the same API key remains active, a later client can miss the duplicate-client warning. Decrement only for clients that were counted.
**Knowledge Base Used:** [Client lifecycle and public API](https://app.greptile.com/posthog-org-19734/-/custom-context/knowledge-base/posthog/posthog-ruby/-/docs/client-lifecycle-and-api.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.This branch has not been deployed
1 participant
💡 Motivation and Context
Several tests could pass without proving the behavior in their descriptions. Examples include a remote-config expectation with no matcher, a fork test that ignored the child's exit status, and a log export test that did not prove the background worker restarted after a fork. The JSON compatibility cases were also missing from normal test discovery.
Cleaning up clients after each example exposed a production bug.
PostHog::MCP::Clienttried to use instance-tracking state on the subclass even though that state belongs toPostHog::Client. This could raise during initialization or shutdown.This PR strengthens payload, retry, timing, flag, MCP, and Rails assertions. It replaces several scheduling sleeps with explicit synchronization and adds cleanup for clients and worker threads. JSON compatibility tests now run in separate processes. An optional OpenTelemetry bundle and CI jobs run the real export and fork tests, which are explicitly pending without those dependencies.
The client tracking calls now use the owning class. Three lifecycle regression cases cover test, synchronous, and asynchronous clients. A patch changeset covers this fix. No public API changes are included.
💚 How did you test it?
git diff --checkpass.05d4864againstorigin/mainreported no actionable findings.Generated coverage data, audit logs, and gem archives remain local and are not included in this PR. The coverage comparison is posted as a separate comment.
📝 Checklist
If releasing new changes
pnpm changesetto generate a changeset fileThe patch changeset was written directly in the repository's changeset format.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Pi performed the audit using file and shell tools, read-only subagent reviews, and the autoreview helper. The work focused on stronger assertions and reliable execution rather than deleting tests or changing unrelated SDK behavior. The MCP lifecycle fix was added after reproducing the failure with regression tests.
Some broader coverage suggestions and ambiguous contracts remain follow-ups. This PR does not claim complete coverage of every SDK behavior. Human review is required.