Harden Agentless Feature Flags EVP delivery by vjfridge · Pull Request #12200 · DataDog/dd-trace-java

vjfridge · 2026-08-12T17:00:37Z

Motivation

We are shipping an agentless, CDN-delivered Feature Flags configuration mode to make customer deployments simpler. Configuration polling is only half of that product path: the exposures and flag-evaluation telemetry produced by the SDK also needs a reliable network path out of the process.

Java therefore needs to auto-discover a compatible local Agent or serverless proxy and fall back to the canonical direct EVP intake when no safe local route exists. That fallback must not duplicate a batch after an ambiguous network failure, leak the API key to a local process, or change the historical Remote Config/Agent-only transport.

This expands the original SDK-identity change into the cohesive Java transport required by the cross-language Agentless EVP contract. Tracks FFLSDK-187 and preserves the FFL-3016 attribution goal.

Changes

Send DD-EVP-ORIGIN: dd-trace-java and DD-EVP-ORIGIN-VERSION: TracerVersion.TRACER_VERSION on both local and direct Feature Flags EVP requests.
Share one process-wide route selector between the exposure and flag-evaluation writers.
In Agentless mode, discover local EVP v4 before v2 and require /info to advertise forwarding support for both SDK identity headers.
Preserve configured Agent URL path prefixes for /info, trace probes, and EVP requests.
Fall back to canonical https://event-platform-intake.<site>/api/v2/{event} with redirects and generic HTTP retries disabled.
Keep direct selection sticky. When neither route is available, allow one bounded local recovery probe per minute.
Replay the current batch only after local HTTP 404/405 or a pre-connect ConnectException. Other local failures switch only future batches when direct intake is available.
Drain and final-flush exposure telemetry during bounded shutdown, including when tracing itself is disabled.

Decisions

The resilient selector is Agentless-only. Remote Config/Agent-only construction remains fixed on local /evp_proxy/v2 without /info discovery or direct credentials; a later flush uses that same fixed route after a failure.
Both event writers share route state so a failure observed by one writer changes the route used by the other without an independent probe.
API keys are added only by the direct intake client. X-Datadog-EVP-Subdomain remains local-only. SDK identity is present on both routes.
Local 403, 429, 5xx, timeouts, resets, and broken pipes are potentially post-send outcomes: the current batch is never replayed. A configured direct route may be used only by later batches.
Direct failures never trigger a local re-probe. Recovery is reserved for the unavailable state and is concurrency-bounded.
DNS suffix and host-length validation is performed before constructing the canonical direct URL.

Validation

Base SHA: 3110972ccf744f304a7cf59973e240f9ccd1a071

Candidate SHA: f1a76966057cb7bf0d9921a8731d016a20fe7575

All commands passed locally:

./gradlew :communication:test --tests datadog.communication.BackendApiFactoryTest --tests datadog.communication.ddagent.DDAgentFeaturesDiscoveryTest
./gradlew :products:feature-flagging:feature-flagging-lib:test --tests com.datadog.featureflag.AgentlessFeatureFlagBackendApiTest --tests com.datadog.featureflag.FeatureFlagBackendApiFactoryTest --tests com.datadog.featureflag.ExposureWriterTests --tests com.datadog.featureflag.FlagEvaluationWriterImplTest
./gradlew :products:feature-flagging:feature-flagging-agent:test --tests com.datadog.featureflag.FeatureFlaggingSystemTest
./gradlew :dd-java-agent:test --tests datadog.trace.agent.TraceDisabledFeatureFlaggingShutdownTest
./gradlew :communication:spotlessCheck :dd-java-agent:spotlessCheck :products:feature-flagging:feature-flagging-lib:spotlessCheck :products:feature-flagging:feature-flagging-agent:spotlessCheck :communication:test :products:feature-flagging:feature-flagging-lib:test :products:feature-flagging:feature-flagging-agent:test :dd-java-agent:shadowJar :dd-trace-api:jar :products:feature-flagging:feature-flagging-api:jar

Added wire/unit coverage for:

Contract	Proof
Local discovery	v4 preference, v2 fallback, both-header capability gate, case-insensitive advertised headers, Agent path prefixes
Credential isolation	API key absent locally; EVP subdomain absent directly; SDK identity present on both routes
Duplicate prevention	same-batch replay only for local 404/405 and pre-connect failure; no replay for 403, 429, 5xx, timeout, or reset; HTTP auto-retry disabled
Shared state and recovery	exposure failure changes the next evaluation route; direct is sticky; unavailable recovery is cooldown- and concurrency-bounded
Compatibility and lifecycle	Remote Config stays fixed on EVP v2; later flush retries the fixed route; queue threshold flush, drain, final flush, bounded close, and tracing-disabled shutdown

No manual system-tests or dogfooding run was claimed for this exact candidate SHA. Those repositories were intentionally left unchanged for this SDK-only PR; the PR's existing system-test workflow is the end-to-end gate.

…ures EVP FeatureFlagEvpContext.from builds the top-level context map shared by both the flagevaluation and exposures EVP writers. Add source.name ("dd-trace-java") and source.version (TracerVersion.TRACER_VERSION) so the SDK identity facets are populated on both EVP streams. This closes the gap noted in the Feature Flag Observability Telemetry Roadmap where the Java server SDK emitted no SDK/tracer name or version on the flag_evaluations EVP stream. Co-Authored-By: Claude <noreply@anthropic.com>

dd-octo-sts · 2026-08-12T17:35:19Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.78 s	14.62 s	[+0.3%; +1.9%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.63 s	13.65 s	[-0.9%; +0.6%] (no difference)
startup:petclinic:appsec:Agent	17.35 s	17.24 s	[-0.2%; +1.4%] (no difference)
startup:petclinic:iast:Agent	16.80 s	17.43 s	[-7.7%; +0.6%] (no difference)
startup:petclinic:profiling:Agent	17.08 s	17.13 s	[-1.4%; +0.9%] (no difference)
startup:petclinic:sca:Agent	17.40 s	16.76 s	[-0.4%; +8.0%] (no difference)
startup:petclinic:tracing:Agent	16.50 s	16.18 s	[-2.3%; +6.2%] (no difference)

Commit: 49eada35 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

…ntext The flagevaluation track schema (logs-backend flagevaluation.conf) declares source.name/source.version as top-level per-event fields, siblings of flag/variant/targeting_key. The previous implementation put them in the batch context envelope alongside service/env/version, which the EVP indexer maps to context.source.* — an undeclared facet that causes the indexer to drop the entire event. Move source to the FlagEvaluationEvent top level (as a nested source object {name,version}) so it lands on the declared source.name/source.version facets. Verified end-to-end via ffe-dogfooding against staging: Java flagevaluation events now index in the staging flag_evaluations data source. Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>

Generated with Claude Code

Send SDK identity headers on both flag evaluation and exposure requests while preserving them across proxy-to-direct fallback. Keep the BackendApi request signature unchanged by configuring headers on feature-flag backend instances.\n\nGenerated with Claude Code

Scope the SDK identity headers to feature-flag HTTP clients so both evaluation and exposure requests retain them across proxy and direct intake without modifying transport implementations.\n\nGenerated with Claude Code

vjfridge · 2026-08-28T02:52:12Z

@codex review

chatgpt-codex-connector · 2026-08-28T02:55:34Z

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 1cb2c70e19

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…-name-and-version-java # Conflicts: # communication/src/main/java/datadog/communication/BackendApiFactory.java

Move the DD-EVP-ORIGIN and DD-EVP-ORIGIN-VERSION header names and the dd-trace-java origin value into EvpProxy, next to the existing subdomain header constant. ProfileUploader and CrashUploader already declare private copies of these names, so the feature-flagging code used string literals as a third copy. Replace the FeatureFlagBackendApiFactory test that asserted the contents of the REQUEST_HEADERS map, which restated the implementation and would pass even if the headers never reached the wire. The field is private again. Cover both transports with MockWebServer assertions instead. The direct intake test uses followRedirects=false to match the feature-flagging caller, so it also covers the redirect-scoped client wrapped by the header interceptor. Removing the interceptor from the direct intake path makes the new test fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The subdomain and Accept-Encoding assertions were added by this branch to a pre-existing test that only checked the request path. Neither relates to SDK identity headers, which evpProxySendsConfiguredRequestHeaders now covers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Share Agentless route state across event writers, preserve Agent prefixes, capability-gate local forwarding, and cover deterministic send-once failover behavior. Environment: Datadog workspace

leoromanovsky · 2026-09-12T03:52:28Z

Superseded by #12477, which carries the same final SDK identity plus Agentless EVP hardening diff as one signed commit directly on the current master SHA. This avoids reviewing the unrelated files exposed by the old branch's merge history; no remote history was rewritten.

datadog-datadog-prod-us1-2

More details

The change keeps the JSON payloads unchanged. It applies the SDK identity headers to both delivery paths without a reportable defect.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit f1a7696 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

vjfridge added tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature comp: telemetry Telemetry type: feature Enhancements and improvements labels Aug 12, 2026

Tighten comments: drop ticket refs and redundant javadoc

8ed8522

This comment has been minimized.

Sign in to view

PerfectSlayer removed the comp: telemetry Telemetry label Aug 13, 2026

vjfridge and others added 4 commits August 13, 2026 13:26

Merge branch 'master' into vickie/ffl-2995-sdk-name-and-version-java

5dc3b91

Merge origin/master into vickie/ffl-2995-sdk-name-and-version-java

b1ba5af

Generated with Claude Code

test(openfeature): verify flag evaluation source metadata

a745b61

Generated with Claude Code

vjfridge changed the title ~~Populate source.name and source.version on flag_evaluations and exposures EVP~~ Add source metadata to flag evaluation events Aug 28, 2026

fix(openfeature): send flag evaluation source headers

b4724e9

Generated with Claude Code

vjfridge changed the title ~~Add source metadata to flag evaluation events~~ Send flag evaluation SDK identity in request headers Aug 28, 2026

vjfridge added 2 commits August 27, 2026 21:34

test(communication): cover backend header fallback

38e9e1b

Generated with Claude Code

vjfridge changed the title ~~Send flag evaluation SDK identity in request headers~~ Send feature flag SDK identity in request headers Aug 28, 2026

refactor(openfeature): inject SDK headers with HTTP client

1cb2c70

Scope the SDK identity headers to feature-flag HTTP clients so both evaluation and exposure requests retain them across proxy and direct intake without modifying transport implementations.\n\nGenerated with Claude Code

vjfridge and others added 3 commits September 9, 2026 16:34

Merge remote-tracking branch 'origin/master' into vickie/ffl-2995-sdk…

8f0cd1a

…-name-and-version-java # Conflicts: # communication/src/main/java/datadog/communication/BackendApiFactory.java

vjfridge marked this pull request as ready for review September 9, 2026 21:02

vjfridge requested review from a team as code owners September 9, 2026 21:02

vjfridge requested review from leoromanovsky and removed request for a team September 9, 2026 21:02

leoromanovsky added 5 commits September 12, 2026 03:19

Harden Feature Flags EVP transport

1a58e69

Drain Feature Flags exposures on shutdown

6b57b41

Keep agent shutdown active when tracing is disabled

d99255f

Validate direct intake DNS site suffixes

84e50e3

Complete Feature Flags EVP route hardening

f1a7696

Share Agentless route state across event writers, preserve Agent prefixes, capability-gate local forwarding, and cover deterministic send-once failover behavior. Environment: Datadog workspace

leoromanovsky requested review from a team as code owners September 12, 2026 03:46

leoromanovsky requested review from bric3, dromanol, jordan-wong, manuel-alvarez-alvarez, mcculls, ojung and randomanderson and removed request for a team September 12, 2026 03:46

leoromanovsky changed the title ~~Send feature flag SDK identity in request headers~~ Harden Agentless Feature Flags EVP delivery Sep 12, 2026

leoromanovsky marked this pull request as draft September 12, 2026 03:47

leoromanovsky mentioned this pull request Sep 12, 2026

Harden Agentless Feature Flags EVP delivery #12477

Draft

leoromanovsky closed this Sep 12, 2026

datadog-datadog-prod-us1-2 Bot reviewed Sep 12, 2026

View reviewed changes

Navigation Menu

Uh oh!

Harden Agentless Feature Flags EVP delivery - #12200

Harden Agentless Feature Flags EVP delivery#12200
vjfridge wants to merge 34 commits into
masterfrom
vickie/ffl-2995-sdk-name-and-version-java

vjfridge commented Aug 12, 2026 •

edited by leoromanovsky

Loading

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

vjfridge commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

Uh oh!

leoromanovsky commented Sep 12, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

Uh oh!

Conversation

vjfridge commented Aug 12, 2026 • edited by leoromanovsky Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Validation

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

vjfridge commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

Uh oh!

leoromanovsky commented Sep 12, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

vjfridge commented Aug 12, 2026 •

edited by leoromanovsky

Loading

dd-octo-sts Bot commented Aug 12, 2026 •

edited

Loading