Harden Agentless Feature Flags EVP delivery

[leoromanovsky]

Motivation

We are shipping an agentless, CDN-delivered Feature Flags configuration mode to make customer deployments simpler. Configuration polling is only half of that product path: the exposures and flag-evaluation telemetry produced by the SDK also needs a reliable network path out of the process.

Java therefore needs to auto-discover a compatible local Agent or serverless proxy and fall back to the canonical direct EVP intake when no safe local route exists. That fallback must not duplicate a batch after an ambiguous network failure, leak the API key to a local process, or change the historical Remote Config/Agent-only transport.

This incorporates and supersedes the identity-only #12200 change in the cohesive Java transport required by the cross-language Agentless EVP contract. Tracks FFLSDK-187 and preserves the FFL-3016 attribution goal.

Changes

• Send DD-EVP-ORIGIN: dd-trace-java and DD-EVP-ORIGIN-VERSION: TracerVersion.TRACER_VERSION on both local and direct Feature Flags EVP requests.
• Share one process-wide route selector between the exposure and flag-evaluation writers.
• In Agentless mode, discover local EVP v4 before v2 and require /info to advertise forwarding support for both SDK identity headers.
• Preserve configured Agent URL path prefixes for /info, trace probes, and EVP requests.
• Fall back to canonical https://event-platform-intake.<site>/api/v2/{event} with redirects and generic HTTP retries disabled.
• Keep direct selection sticky. When neither route is available, allow one bounded local recovery probe per minute.
• Replay the current batch only after local HTTP 404/405 or a pre-connect ConnectException. Other local failures switch only future batches when direct intake is available.
• Drain and final-flush exposure telemetry during bounded shutdown, including when tracing itself is disabled.

Decisions

• The resilient selector is Agentless-only. Remote Config/Agent-only construction remains fixed on local /evp_proxy/v2 without /info discovery or direct credentials; a later flush uses that same fixed route after a failure.
• Both event writers share route state so a failure observed by one writer changes the route used by the other without an independent probe.
• API keys are added only by the direct intake client. X-Datadog-EVP-Subdomain remains local-only. SDK identity is present on both routes.
• Local 403, 429, 5xx, timeouts, resets, and broken pipes are potentially post-send outcomes: the current batch is never replayed. A configured direct route may be used only by later batches.
• Direct failures never trigger a local re-probe. Recovery is reserved for the unavailable state and is concurrency-bounded.
• DNS suffix and host-length validation is performed before constructing the canonical direct URL.

Validation

Base SHA: 3110972ccf744f304a7cf59973e240f9ccd1a071

Candidate SHA: e20a9d77c333ddb0c0444835f23fafba70952373

All commands passed locally:

./gradlew :communication:test --tests datadog.communication.BackendApiFactoryTest --tests datadog.communication.ddagent.DDAgentFeaturesDiscoveryTest
./gradlew :products:feature-flagging:feature-flagging-lib:test --tests com.datadog.featureflag.AgentlessFeatureFlagBackendApiTest --tests com.datadog.featureflag.FeatureFlagBackendApiFactoryTest --tests com.datadog.featureflag.ExposureWriterTests --tests com.datadog.featureflag.FlagEvaluationWriterImplTest
./gradlew :products:feature-flagging:feature-flagging-agent:test --tests com.datadog.featureflag.FeatureFlaggingSystemTest
./gradlew :dd-java-agent:test --tests datadog.trace.agent.TraceDisabledFeatureFlaggingShutdownTest
./gradlew :products:feature-flagging:feature-flagging-lib:spotlessCheck :products:feature-flagging:feature-flagging-lib:test --tests com.datadog.featureflag.AgentlessFeatureFlagBackendApiTest --tests com.datadog.featureflag.FeatureFlagRouteSelectorTest --tests com.datadog.featureflag.ExposureWriterTests
./gradlew :products:feature-flagging:feature-flagging-lib:jacocoTestCoverageVerification -PcheckCoverage -PtestJvm=8
./gradlew :communication:spotlessCheck :dd-java-agent:spotlessCheck :products:feature-flagging:feature-flagging-lib:spotlessCheck :products:feature-flagging:feature-flagging-agent:spotlessCheck :communication:test :products:feature-flagging:feature-flagging-lib:test :products:feature-flagging:feature-flagging-agent:test :dd-java-agent:test --tests datadog.trace.agent.TraceDisabledFeatureFlaggingShutdownTest :dd-java-agent:shadowJar :dd-trace-api:jar :products:feature-flagging:feature-flagging-api:jar

The final affected-module verification completed successfully with 1,112 Gradle tasks. The Java 8 JaCoCo verification also passes the repository's 90% changed-class coverage gate.

Added wire/unit coverage for:

Contract	Proof
Local discovery	v4 preference, v2 fallback, both-header capability gate, case-insensitive advertised headers, Agent path prefixes
Credential isolation	API key absent locally; EVP subdomain absent directly; SDK identity present on both routes
Duplicate prevention	same-batch replay only for local 404/405 and pre-connect failure; no replay for 403, 429, 5xx, timeout, or reset; HTTP auto-retry disabled
Shared state and recovery	exposure failure changes the next evaluation route; writer-specific clients safely follow shared state; direct is sticky; unavailable recovery is cooldown- and concurrency-bounded
Compatibility and lifecycle	Remote Config stays fixed on EVP v2; later flush retries the fixed route; queue threshold flush, drain, final flush, missing-backend termination, bounded close, and tracing-disabled shutdown

Remote CI snapshot for exact candidate head e20a9d77c333ddb0c0444835f23fafba70952373:

• 530 checks pass. Four concrete GitLab jobs are red; DDCI Status and dd-gitlab/default-pipeline are aggregate failures, and devflow/mergegate remains pending downstream of review/status.
• build_tests: [:smokeTest, smoke, -Xms256M -Xmx1024M] and dependent check_smoke 1/4: CI Visibility classifies both as retriable platform_error dependency failures. The internal Maven repository returned HTTP 503 while resolving metadata for com.datadoghq:dd-trace-api.
• test_inst: [17, 7/8]: CI Visibility classifies the Java heap-space exhaustion as a retriable platform_error infrastructure failure.
• muzzle: [7/8]: the unchanged Robolectric instrumentation reports missing org.robolectric.versioning.AndroidVersions classes. The identical shard failure is present on multiple unrelated Java branches in the same seven-day window, so this is shared repository CI breakage rather than an EVP transport regression.

No Feature Flags or changed transport test is failing in CI. The exact-candidate spring-boot system-test artifact passed direct delivery 3/3, serverless-init delivery 2/2, and activation/manifest checks 37/37 in DataDog/system-tests#7702.

[linear-code]

FFLSDK-187

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 72.16%
• Overall Coverage: 58.23% (-0.92%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e20a9d7 | Docs | View more details | Give us feedback!}

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.78 s	14.77 s	[-0.8%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent	13.62 s	13.66 s	[-1.1%; +0.5%] (no difference)
startup:petclinic:appsec:Agent	17.60 s	17.49 s	[-0.4%; +1.6%] (no difference)
startup:petclinic:iast:Agent	17.42 s	17.62 s	[-1.9%; -0.3%] (maybe better)
startup:petclinic:profiling:Agent	17.38 s	16.76 s	[-1.0%; +8.4%] (no difference)
startup:petclinic:sca:Agent	17.64 s	17.53 s	[-0.1%; +1.4%] (no difference)
startup:petclinic:tracing:Agent	16.61 s	16.72 s	[-1.8%; +0.5%] (no difference)

Commit: e20a9d77 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.