[SDTEST-3873] Add internal telemetry metrics

[anmarchenko]

What

• Added a narrow, metrics-only internal/telemetry package based on the CI Visibility telemetry implementation in dd-trace-go.
  • Supports count and distribution metrics with deterministic tag handling.
  • Aggregates concurrent submissions and restores buffered values after failed or cancelled flushes.
  • Sends count and distribution payloads together using the telemetry v2 message-batch request type when both are present.
  • Sends no app-started, lifecycle, configuration, dependency, integration, log, or heartbeat events.
• Added the telemetry v2 request envelope and headers with a per-process runtime ID, sequence IDs, host/runtime metadata, the configured ddtest service name, DD_ENV, the ddtest build version, and the constant language name ddtest.
• Added Agent and agentless telemetry delivery.
  • Agent mode uses /telemetry/proxy/api/v2/apmtelemetry and supports HTTP(S) and Unix-socket Agent URLs.
  • Agentless mode uses the instrumentation telemetry intake for DD_SITE, or DD_CIVISIBILITY_AGENTLESS_URL when a test intake override is configured.
  • API keys are attached only for agentless requests.
• Extracted shared Datadog connection discovery and HTTP transport construction into internal/httptransport, then reused it from both Test Optimization and telemetry to keep Agent URL, Unix socket, site, API key, and agentless test URL handling consistent.
• Created one telemetry client per plan or test command, passed the same client through the runner, planner, Test Optimization API transport, and Git command runner, and flushed it after the command completes. Client creation and flush failures remain best-effort and do not replace the command result.
• Added CI Visibility metrics matching the current dd-trace-go names and tags where they apply to ddtest:
  • Search-commits, object-pack, settings, skippable-tests, known-tests, and test-management request counts, errors, durations, response sizes, and response item counts.
  • Settings response feature flags.
  • Aggregate itr_skipped counts tagged as event_type:test or event_type:suite. ddtest does not emit itr_unskippable or itr_forced_run.
  • Low-level Git command counts, errors, and durations for the Git operations used by repository upload, with canonical command and exit-code tags. Existing package-level Git functions retain their non-telemetry behavior.
• Added unit and integration-style coverage for metric aggregation, batching, request metadata, Agent/agentless routing, retries, concurrency, API metrics, planner counts, Git metrics, and end-to-end dependency wiring.

Why

ddtest performs Test Optimization API requests, repository uploads, TIA planning, and supporting Git operations in its own process. Those operations need internal CI Visibility metrics so their volume, latency, response sizes, and failures can be monitored independently of the language tracer running the tests.

The implementation intentionally copies only the metrics functionality needed by ddtest. It uses the optional telemetry message-batch envelope to avoid duplicating the common request header when flushing both count and distribution payloads, while excluding unrelated application lifecycle events.

Sharing connection discovery with the existing Test Optimization transport also prevents the telemetry and API clients from resolving Agent, Unix-socket, site, API-key, and test-intake configuration differently.

E2E testing

• Ran make test successfully.
• Ran make lint successfully with zero issues.
• Manual scenario:
  1. Start a local mock intake that implements the Test Optimization settings/skippables responses and records telemetry POST requests.
  2. Set DD_CIVISIBILITY_AGENTLESS_ENABLED=true, DD_API_KEY=test, DD_CIVISIBILITY_AGENTLESS_URL=http://127.0.0.1:<port>, DD_SERVICE=<service>, and DD_ENV=e2e.
  3. Run go run . plan in a supported test project.
  4. Verify a request reaches /api/v2/apmtelemetry, uses language_name: ddtest and the configured service, contains only generate-metrics/distributions payloads (wrapped in message-batch when both exist), and includes the expected API, ITR, and Git metrics.

[datadog-datadog-prod-us1-2]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 91.05%
• Overall Coverage: 87.88% (+0.43%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e2362f3 | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a58d07e15

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-datadog-prod-us1-2]

More details

The telemetry transport, aggregation and restoration paths behaved correctly across Agent and agentless routing, retries, cancellation, malformed configuration, concurrent submissions, and realistic metric payload shapes. No production-impacting regression was reproducible; lint remains unverified because the installed golangci-lint binary targets Go 1.24 while the repository targets Go 1.26.3.

Was this helpful? React 👍 or 👎

_{📊 Validated against 10 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 6a58d07 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[anmarchenko]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 16e13e7291

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[anmarchenko]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e2362f36b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[anmarchenko]

E2E Test Report: ISSUES FOUND ❌

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

• Method: Local E2E testing with the RuboCop and Vagrant playgrounds, mockdog, and the configured Datadog EU target
• Revision tested: e2362f36b3c52c65b12b7ae26a825c2925d05182
• Dependency: ddtest=anmarchenko/add-internal-telemetry-metrics
• Debug logging: enabled for every run

Results

Check	Status	Evidence
Build and inject PR branch	✅	Crook built the PR revision successfully for every run
Full ddtest run completes	✅	2 RSpec examples: 1 passed, 1 skipped by TIA, 0 failures
Agentless telemetry routing	✅	One /api/v2/apmtelemetry request with API key
Agent telemetry routing	✅	/telemetry/proxy/api/v2/apmtelemetry, without API key
Metrics-only batching	✅	One message-batch containing only generate-metrics and distributions
Application/host metadata	✅	Service, environment, build version, runtime UUID, sequence ID, language, and host metadata were populated
CI Visibility metrics	✅	Settings, skippable-tests, Git command, latency/size, and itr_skipped event_type:test metrics matched the exercised behavior
TIA behavior and spans	✅	Mockdog recorded a passing ITR session with one passing test and one TIA-skipped test
Live Test Optimization API compatibility	✅	EU settings and test-management responses parsed successfully; settings response shape matched mockdog
language_name: ddtest downstream behavior	✅	Confirmed working by the PR owner after testing
Telemetry opt-out	❌	ddtest still posted telemetry with DD_INSTRUMENTATION_TELEMETRY_ENABLED=false
Distribution payload schema	❌	The distributions payload omitted the mandatory top-level namespace property
Telemetry session correlation header	❌	DD-Session-ID was absent even though a body runtime_id was present

Issues Found

1. DD_INSTRUMENTATION_TELEMETRY_ENABLED=false is ignored.

   • Reproduced with a focused RuboCop planner run.
   • Mockdog still received a ddtest message-batch.
   • Resolution: Not resolved in the tested revision.

2. The distributions sub-payload omits its top-level namespace.

   • Every series had namespace: civisibility, but the payload object contained only series.
   • The telemetry v2 distributions schema marks the payload-level property mandatory; current dd-trace-go also serializes it.
   • Resolution: Not resolved in the tested revision.

3. DD-Session-ID is missing.

   • The body contained a stable UUID runtime_id, but no matching session header was sent.
   • Current Go, JavaScript, Python, Ruby, Java, and .NET telemetry implementations send this correlation header.
   • Resolution: Not resolved in the tested revision.

Verification

• Mockdog raw payloads verified endpoint paths, headers, envelope structure, metadata, counts, distributions, and tags.
• Mockdog spans verified the full runner result: 1 pass, 1 TIA skip, 0 failures.
• A planner run against the real EU backend verified live settings and test-management response compatibility after the shared transport refactor.
• The PR owner confirmed that language_name: ddtest works downstream.
• A Datadog Test Visibility UI session was not expected from the plan-only EU run; the full test execution intentionally targeted mockdog.

Test Methodology

1. Refreshed all Test Optimization tracer source clones and compared the new telemetry implementation with current Go, JavaScript, Python, and Ruby clients.
2. Ran a focused RuboCop ddtest plan against mockdog with one test-level TIA skip.
3. Ran ddtest run to completion on the same fixture and inspected raw telemetry plus parsed test/session spans.
4. Repeated planning in Agent mode to verify proxy routing and API-key isolation.
5. Ran a Vagrant unskippable-marker scenario to validate final skip accounting.
6. Set DD_INSTRUMENTATION_TELEMETRY_ENABLED=false and confirmed the opt-out regression.
7. Ran the focused planner against the configured EU backend and compared cached response shapes with mockdog.

---

This E2E test was performed by Shepherd — autonomous QA for Datadog Test Optimization.