Emit gen_ai.* attributes on APM spans by mz1119 · Pull Request #12450 · DataDog/dd-trace-java

mz1119 · 2026-09-10T15:02:09Z

What Does This Do

Emits the scalar gen_ai.* attributes onto APM spans, making them indexed APM tags:

meta: gen_ai.operation.name, gen_ai.request.model, gen_ai.provider.name, gen_ai.application.name, gen_ai.conversation.id
metrics: gen_ai.usage.{input,output,total,cache_read_input,cache_write_input,reasoning_output}_tokens

GenAiApmTags reads the values back off the span's _ml_obs_tag.* / _ml_obs_metric.* keys at finish time and is called from both emission sites: manual SDK spans (DDLLMObsSpan.finish) and auto-instrumentation (OpenAiDecorator.doBeforeFinish). Since LLMObsSpanMapper only serializes _ml_obs_* keys, the unprefixed tags land on the APM span only.

Message bodies (input, output, tool definitions, retrieval documents) stay off the APM span and keep coming from the LLMObs track. Port of DataDog/dd-trace-py#20083 and DataDog/dd-trace-js#10261.

Motivation

The APM trace UI merges these keys from the LLMObs track client-side, so the values render but aren't indexed. You can't filter, facet, or monitor on model, provider, or token usage in APM.

Additional Notes

llm and embedding spans always get model and provider, falling back to custom (matching what the LLMObs event already writes), and are the only kinds that get gen_ai.usage.*. Other kinds get model fields only when explicitly set, since their metrics would be misleading under a gen_ai.usage.* key.

Emission runs ahead of span.finish() while the span is still mutable, in its own try/catch so a failure can't cost the LLMObs event.

The openai-java instrumentation keeps tracing with LLM Observability disabled, so applyWithoutLlmObs emits what is resolvable there: operation (from openai.request.endpoint), model (from openai.response.model, falling back to the request model), provider, and ml_app. Token usage and conversation id are not computed on that path and stay unreported, which is the one place this diverges from the Python PR. Java has no _dd.llmobs.* shadow-tag equivalent to read them from.

Verified against live OpenAI calls with the local agent build. With LLM Observability enabled:

[apm span] openai.request / createResponse
  gen_ai.application.name = java-genai-test
  gen_ai.conversation.id = session-abc-123
  gen_ai.operation.name = llm
  gen_ai.provider.name = openai
  gen_ai.request.model = gpt-4.1-mini-2025-04-14
  gen_ai.usage.input_tokens = 13.0
  gen_ai.usage.output_tokens = 10.0
  gen_ai.usage.total_tokens = 23.0

With it disabled:

[apm span] openai.request / createResponse
  gen_ai.application.name = java-genai-test
  gen_ai.operation.name = llm
  gen_ai.provider.name = openai
  gen_ai.request.model = gpt-4.1-mini-2025-04-14

🤖 Generated with Claude Code

Write the scalar gen_ai.* subset of an LLM Observability span onto the APM span, so model, provider, application, conversation and token usage become indexed, searchable APM tags. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dd-octo-sts · 2026-09-10T15:28:47Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.85 s	14.68 s	[-0.0%; +2.3%] (no difference)
startup:insecure-bank:tracing:Agent	13.60 s	13.62 s	[-1.0%; +0.8%] (no difference)
startup:petclinic:appsec:Agent	17.61 s	17.42 s	[+0.3%; +1.9%] (maybe worse)
startup:petclinic:iast:Agent	17.36 s	17.53 s	[-1.7%; -0.3%] (maybe better)
startup:petclinic:profiling:Agent	17.22 s	17.35 s	[-2.1%; +0.5%] (no difference)
startup:petclinic:sca:Agent	17.51 s	17.44 s	[-0.4%; +1.2%] (no difference)
startup:petclinic:tracing:Agent	16.48 s	16.72 s	[-2.3%; -0.6%] (maybe better)

Commit: 324b5bbd · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

mz1119 · 2026-09-10T18:26:33Z

@codex review

chatgpt-codex-connector · 2026-09-10T18:26:52Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-11T17:28:18.683154Z	`1527dca`	Draft marked ready
🔒 Security Review	✅ Completed 2026-09-11T17:29:41.733301Z	`1527dca`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-09-10T18:31:01Z

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: f1d46cfe83

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector · 2026-09-10T18:31:04Z

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: f1d46cfe83

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The openai-java instrumentation keeps tracing when LLM Observability is disabled, and the operation, model, provider and ml_app are all known on that path. Token usage and conversation id are not computed there, so they stay unreported. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The repository does not accept new .groovy files. The module already has a JUnit forked-test base for LLMObs configurations, so the coverage moves there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

datadog-datadog-us1-prod

More details

The finish-time mapping reads only scalar LLM Observability tags. It keeps message bodies off APM spans and limits usage metrics to LLM and embedding spans.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 1527dca · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

ncybul · 2026-09-11T18:16:56Z

@codex

chatgpt-codex-connector · 2026-09-11T18:20:31Z

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: faac2ec6f8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

ncybul · 2026-09-11T18:27:39Z

+   * Writes the subset available with LLM Observability disabled. Token usage and conversation id
+   * are never computed on that path, so they are left out.
+   */
+  public static void applyWithoutLlmObs(


Is it possible to have one method that is called for all situations regardless of whether llmobs is enabled (although I think this might be a bit more annoying in Java since there are no optional parameters)?

Feel free to disregard if it ends up making things more complex.

ncybul · 2026-09-11T18:37:10Z

+    return value == null || value.isEmpty() ? null : value;
+  }
+
+  private static String stringTag(AgentSpan span, String key) {


This seems to be repeated in the OpenAiDecorator.java file. Is there a way to avoid this duplication?

ncybul · 2026-09-11T18:40:28Z

+            "gen_ai.request.model" String
+            "gen_ai.provider.name" "openai"
+            "gen_ai.application.name" String
+            if (!isStreaming) {


Do we not collect token usage for streamed Open AI spans?

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Emit gen_ai.* attributes on APM spans

f1d46cf

Write the scalar gen_ai.* subset of an LLM Observability span onto the APM span, so model, provider, application, conversation and token usage become indexed, searchable APM tags. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

mz1119 added type: feature Enhancements and improvements comp: mlobs ML Observability (LLMObs) tag: ai generated Largely based on code generated by an AI or LLM labels Sep 10, 2026

This comment has been minimized.

Sign in to view

mz1119 and others added 2 commits September 11, 2026 10:26

Rewrite the LLMObs-disabled openai test in Java

1527dca

The repository does not accept new .groovy files. The module already has a JUnit forked-test base for LLMObs configurations, so the coverage moves there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

mz1119 marked this pull request as ready for review September 11, 2026 17:23

mz1119 requested review from a team as code owners September 11, 2026 17:23

mz1119 requested review from vandonr and removed request for a team September 11, 2026 17:23

datadog-datadog-us1-prod Bot reviewed Sep 11, 2026

View reviewed changes

Trim comments and docstrings

faac2ec

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

mz1119 mentioned this pull request Sep 11, 2026

feat(llmobs): emit gen_ai.* APM tags when LLMObs is disabled DataDog/dd-trace-js#10300

Draft

ncybul approved these changes Sep 11, 2026

View reviewed changes

Address review: single apply method and shared tag reader

324b5bb

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Navigation Menu

Uh oh!

Emit gen_ai.* attributes on APM spans - #12450

Emit gen_ai.* attributes on APM spans#12450
mz1119 wants to merge 5 commits into
masterfrom
max.zhang/llmobs-gen-ai-apm-tags

mz1119 commented Sep 10, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

mz1119 commented Sep 10, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 10, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 10, 2026

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Uh oh!

ncybul commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026

Uh oh!

ncybul Sep 11, 2026

Uh oh!

ncybul Sep 11, 2026

Uh oh!

ncybul Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mz1119 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

mz1119 commented Sep 10, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Sep 10, 2026

🛡️ Codex Security Review · Automatically triggered

Uh oh!

chatgpt-codex-connector Bot commented Sep 10, 2026

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Choose a reason for hiding this comment

Uh oh!

ncybul commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026

Uh oh!

ncybul Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

ncybul Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

ncybul Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mz1119 commented Sep 10, 2026 •

edited

Loading

dd-octo-sts Bot commented Sep 10, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 10, 2026 •

edited

Loading