Skip to content

feat: surface experiment metadata on flags and add event tracking - #94

Draft
Zaimwa9 wants to merge 4 commits into
mainfrom
feat/experimentation
Draft

Zaimwa9 wants to merge 4 commits into
mainfrom
feat/experimentation

Conversation

@Zaimwa9

@Zaimwa9 Zaimwa9 commented Sep 18, 2026

Copy link
Copy Markdown

Thanks for submitting a PR! Please check the boxes below:

  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature.
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

Brings the Flutter SDK to parity with JS (Flagsmith/flagsmith-js-client#420) and Python (Flagsmith/flagsmith-python-client#249) for experimentation. Core sends metadata.experiment { id, name, in_experiment } on /identities flags while the feature's experiment is running (Flagsmith/flagsmith#8532).

  • Flag.variant, Flag.reason and Flag.experiment { id, name, inExperiment }, lifted from metadata.experiment. null on older servers and on /flags. Round-trips through persistent storage.
  • Events pipeline behind FlagsmithConfig(enableEvents: true): EventProcessor batches to {eventsURI}v1/events, flushes every eventsFlushInterval (10 s) or at eventsMaxBuffer (1000), dedupes exposures per flush window, retries a failed POST once then drops. Never throws.
  • getExperimentFlag resolves the flag and records one $flag_exposure with the variant as value and metadata.experiment_id, only when flag.experiment.inExperiment is true. Skips are logged.
  • trackEvent, trackExposureEvent, flushEvents; close() flushes best-effort. Event names starting with $ are rejected.
  • README "Experiments" section.

Release after Core (Flagsmith/flagsmith#8532) and Edge (Flagsmith/edge-api#719) are live. Against older servers no flag carries experiment, so no exposure is ever recorded.

How did you test this code?

  • flutter analyze clean, flutter test 162 passing: model parsing and round-trip, getExperimentFlag gating (enrolled, not enrolled, no metadata, disabled, missing, no identity, events off), dedupe, flush on interval / max buffer / flushEvents / close, retry then drop.
  • Production parity: 300 identities evaluated through the SDK and through curl against Edge, all 68 flags per identity compared on enabled, value, variant, reason and experiment metadata, zero differences. Variant split matched the experiment's 75% rollout and 30/70 weights within sampling noise.
  • Demo app on the iOS Simulator against a production environment: exposures and conversions arrive in the experiment dashboard.

@Zaimwa9

Zaimwa9 commented Sep 18, 2026

Copy link
Copy Markdown
Author

@themis-blindfold review

Comment thread lib/src/flagsmith_client.dart Outdated
cachedUser = identity;
}
final flags = await getFeatureFlags(
user: identity, traits: traits, reload: reload ?? false);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Major · ⚡ Quick win

Fetch flags for an explicitly supplied identity before exposing it.

Observed: This defaults reload to false, so it reads the shared feature-name store after setting cachedUser to user; the stored flag and experiment data can belong to a previous identity.

Predicted: Calling getExperimentFlag(..., user: B) after A populated storage would return A's assignment and send the exposure as B, corrupting the displayed variant and experiment attribution. Default to an identity fetch whenever user is supplied unless the caller explicitly sets reload, and add a two-identity regression test.

Suggested change
user: identity, traits: traits, reload: reload ?? false);
user: identity, traits: traits, reload: reload ?? (user != null));

@themis-blindfold

Copy link
Copy Markdown

⚖️ Themis review: 🟠 Fix before merge

The experiment metadata and event batching largely align with the established SDK behaviour, but the explicit-user convenience path can reuse a prior identity's stored flag before recording an exposure. That would misattribute an experiment assignment when an application switches users. CI is still running; only the title check has completed.

Area Score
🎯 Correctness 3/5
🧪 Test coverage 3/5
📐 Code quality 4/5
🚀 Product impact 4/5

🟠 Majors

  • lib/src/flagsmith_client.dart:374 — an explicit identity can be exposed against the previous identity's cached experiment assignment.
📝 Walkthrough
  • Flag model - surfaces variant, reason, and optional experiment metadata while preserving it through storage.
  • Event pipeline - batches conversions and exposures, deduplicates exposure events, and retries failed posts once.
  • Client API - adds experiment reads and event tracking, including best-effort flushing at shutdown.
  • Documentation and QA - describes the experimental workflow and adds mock-backed smoke coverage.
🧪 How to verify
  1. Fetch experiment flags for identity A, then call getExperimentFlag for identity B without an explicit reload; confirm the request fetches B before returning or exposing a flag.
  2. Use identities assigned to different variants and confirm both the returned variant and posted exposure identifier match B.
  3. Run flutter test test/fg/flagsmith_experiments_test.dart test/core/models/flag_test.dart.
  4. Run flutter analyze.
    Automate: add a two-identity storage regression test for the default explicit-user path.

Product take: Experiment tracking is a solid capability improvement, but incorrect assignment attribution would undermine experiment results for apps that switch users.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

Experiments need the right person in the right bucket · reviewed at 56cd56c

@Zaimwa9
Zaimwa9 force-pushed the feat/experimentation branch from 56cd56c to a1cdd64 Compare September 18, 2026 15:14
@Zaimwa9

Zaimwa9 commented Sep 18, 2026

Copy link
Copy Markdown
Author

@themis-blindfold review

Comment thread README.md Outdated
@themis-blindfold

Copy link
Copy Markdown

⚖️ Themis review: 🧹 Ship it, nits inside

The experiment metadata and opt-in event pipeline match the intended behaviour, including identity-specific reloads and experiment-aware exposure deduplication. One README snippet needs a null check before it can be copied into a null-safe Dart app. CI is still running; the local Flutter SDK was unavailable for the targeted checks.

Area Score
🎯 Correctness 4/5
🧪 Test coverage 4/5
📐 Code quality 4/5
🚀 Product impact 4/5

🧹 Nits

  • README.md: manual-exposure example needs to handle the nullable flag result.
📝 Walkthrough
  • Flag model - surfaces variant, reason, and optional experiment metadata while preserving it through storage.
  • Experiment reads - fetches explicitly supplied identities and gates automatic exposures on enrolled, enabled flags.
  • Event delivery - batches events to the events endpoint, deduplicates exposures per flush window, and retries one failed post.
  • Documentation - explains configuration, exposure recording, conversion events, and flushing.
🧪 How to verify
  1. Run flutter analyze.
  2. Run flutter test test/core/models/flag_test.dart test/fg/flagsmith_experiments_test.dart.
  3. Copy the README exposure example into a null-safe Flutter app and confirm it analyses and only emits an exposure for an enrolled identity.
  4. Point eventsURI at a test endpoint, force one failed POST, and confirm the batch retries exactly once.
    Automate: add a checked documentation snippet for the manual-exposure example.

Product take: A solid experimentation capability: clients can report enrolment-aware exposures and conversions without changing ordinary flag reads.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

A small null check, and the experiment instructions are ready for their close-up · reviewed at a1cdd64

@Zaimwa9

Zaimwa9 commented Sep 18, 2026

Copy link
Copy Markdown
Author

@themis-blindfold review

}

/// Never throws.
Future<void> flush() async {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Major · ⚡ Quick win

flushEvents() can complete before an automatic upload finishes.

Observed: A max-buffer or timer flush clears _buffer before awaiting _postBatch; a following flushEvents() therefore sees an empty buffer and returns while that POST is still in flight.

Predicted: Code that awaits flushEvents() during teardown after an automatic flush could terminate before the event request completes, despite the documented completion guarantee. Track in-flight flush futures and make flush() await them as well as any newly buffered batch; add a delayed-request regression test for both max-buffer and timer flushes.

@themis-blindfold

Copy link
Copy Markdown

⚖️ Themis review: 🟠 Fix before merge

The experiment metadata and event pipeline are broadly covered, and the completed publish, analysis, and test checks passed. One lifecycle race breaks the documented guarantee that awaiting flushEvents() waits for the event POST: an automatic flush empties the buffer before its upload completes, so a subsequent explicit flush returns immediately.

Area Score
🎯 Correctness 3/5
🧪 Test coverage 4/5
📐 Code quality 4/5
🚀 Product impact 4/5

🟠 Majors

⚖️ Acknowledged

  • Explicit-identity experiment reads now fetch that identity before recording an exposure — thread resolved by @Zaimwa9
  • Manual-exposure example dereferences the nullable flag result — thread resolved by @Zaimwa9
📝 Walkthrough
  • Flag model - exposes variant, reason, and typed experiment metadata while preserving it through storage serialization.
  • Client API - adds gated experiment exposure reads and public conversion/exposure tracking methods.
  • Event delivery - batches event payloads, deduplicates exposures, and retries one failed request; explicit flushing does not currently join an already-running automatic batch.
  • Documentation and tests - documents the opt-in flow and exercises enrollment, event shaping, retries, buffering, and persistence.
🧪 How to verify
  1. Configure eventsMaxBuffer: 1 and delay the events endpoint response.
  2. Call trackEvent, then immediately await flushEvents().
  3. Assert that flushEvents() does not complete until the delayed POST completes.
  4. Repeat with a timer-triggered flush already in progress.
    Automate: add both cases to test/fg/flagsmith_experiments_test.dart using the existing delayed HTTP adapter.

Product take: This is a solid experimentation capability, but teardown-sensitive apps could lose attribution by trusting the documented flush completion signal. That makes the lifecycle guarantee worth fixing before release.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

Events are patient; app teardown generally is not. · reviewed at a1cdd64

@Flagsmith Flagsmith deleted a comment from themis-blindfold Bot Sep 18, 2026
@Flagsmith Flagsmith deleted a comment from themis-blindfold Bot Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant