Skip to content

Commit de558fe

Browse files
D-K-PD-K-P
committed
docs(ai): add LLM observability page
Document capturing Vercel AI SDK calls as spans in the run trace: turning it on per call, what each span inspector tab shows, linking a call to its prompt version, and querying usage across runs with TRQL.
1 parent d3d0e08 commit de558fe

2 files changed

Lines changed: 166 additions & 0 deletions

File tree

‎docs/ai/observability.mdx‎

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
---
2+
title: "LLM observability"
3+
sidebarTitle: "LLM observability"
4+
description: "Capture Vercel AI SDK calls in a task as spans in the run trace, with model, token usage, cost, and latency. Opt in per call, link calls to prompt versions, and query usage across runs."
5+
---
6+
7+
**LLM observability turns a Vercel AI SDK call inside a task into its own span in the run trace, next to your logs and other spans.** Each span carries the model, provider, input, output, and total token counts, cost, and latency, so you can see what each generation did and what it cost without leaving the run.
8+
9+
Everything shows up inline in the run trace you already use to debug runs. There is no separate product and no dashboard to set up.
10+
11+
<Note>
12+
Observability is opt-in per call and only covers [Vercel AI SDK](https://ai-sdk.dev) functions (`generateText`, `streamText`, `generateObject`). Calls you make with a raw `fetch`, a provider's own SDK, or any other HTTP client are not captured automatically.
13+
</Note>
14+
15+
## Turn it on
16+
17+
Set `experimental_telemetry: { isEnabled: true }` on the AI SDK call. There is nothing to install for AI SDK 6, and nothing to configure on the Trigger.dev side.
18+
19+
```ts /trigger/summarize.ts
20+
import { task } from "@trigger.dev/sdk";
21+
import { generateText } from "ai";
22+
import { openai } from "@ai-sdk/openai";
23+
24+
export const summarize = task({
25+
id: "summarize",
26+
run: async (payload: { text: string }) => {
27+
const result = await generateText({
28+
model: openai("gpt-4o"),
29+
prompt: `Summarize the following text:\n\n${payload.text}`,
30+
experimental_telemetry: { isEnabled: true },
31+
});
32+
33+
return { summary: result.text };
34+
},
35+
});
36+
```
37+
38+
Trigger the task and open the run. The `generateText` call appears as a span in the trace. `streamText` and `generateObject` work the same way: add the same `experimental_telemetry` flag to each call you want captured.
39+
40+
<Note>
41+
On AI SDK 7, span emission moved out of the `ai` core into the `@ai-sdk/otel` adapter. Install `@ai-sdk/otel` and Trigger.dev registers it for you at run start. AI SDK 6 emits spans from `ai` directly, so no extra package is needed.
42+
</Note>
43+
44+
## What each span shows
45+
46+
Open an AI generation span in the run trace to get a dedicated inspector with three tabs:
47+
48+
- **Overview**: model, provider, token usage, cost, and a preview of the input and output.
49+
- **Messages**: the full message thread, including the system prompt and any tool results.
50+
- **Tools**: the tool definitions passed to the model, plus every tool call the model made with its arguments.
51+
52+
A fourth **Prompt** tab appears when the call is linked to an [AI Prompt](/ai/prompts) (see below).
53+
54+
## Link a call to its prompt
55+
56+
If you manage prompts with [AI Prompts](/ai/prompts), resolve the prompt and spread `toAISDKTelemetry()` into the call. This sets `experimental_telemetry` for you and links the span back to the exact prompt version that produced it.
57+
58+
```ts /trigger/support.ts
59+
import { task, prompts } from "@trigger.dev/sdk";
60+
import { generateText } from "ai";
61+
import { openai } from "@ai-sdk/openai";
62+
import type { supportPrompt } from "./prompts";
63+
64+
export const handleSupport = task({
65+
id: "handle-support",
66+
run: async (payload: { name: string; plan: string; issue: string }) => {
67+
const resolved = await prompts.resolve<typeof supportPrompt>("customer-support", {
68+
customerName: payload.name,
69+
plan: payload.plan,
70+
issue: payload.issue,
71+
});
72+
73+
const result = await generateText({
74+
model: openai(resolved.model ?? "gpt-4o"),
75+
system: resolved.text,
76+
prompt: payload.issue,
77+
...resolved.toAISDKTelemetry(),
78+
});
79+
80+
return { response: result.text };
81+
},
82+
});
83+
```
84+
85+
The span's **Prompt** tab now shows the linked template, its version, and the input variables the prompt was resolved with.
86+
87+
Pass custom attributes to `toAISDKTelemetry()` to tag the span with your own metadata:
88+
89+
```ts
90+
const result = await generateText({
91+
model: openai(resolved.model ?? "gpt-4o"),
92+
system: resolved.text,
93+
prompt: payload.issue,
94+
...resolved.toAISDKTelemetry({
95+
"task.type": "summarization",
96+
"customer.tier": "enterprise",
97+
}),
98+
});
99+
```
100+
101+
Custom attributes are stored on the span's `metadata`, so you can filter or group by them in TRQL, for example `metadata['task.type']`.
102+
103+
<Note>
104+
When you build an agent with `chat.agent()`, `chat.toStreamTextOptions()` already sets `experimental_telemetry` for you, so generations inside a chat are captured without adding the flag by hand. See [Prompts](/ai/prompts#using-with-chatagent).
105+
</Note>
106+
107+
## Query usage across runs
108+
109+
Every captured generation is also written to the `llm_metrics` table, which you can query with [TRQL](/observability/query). This lets you aggregate token usage, cost, and latency across many runs rather than inspecting one span at a time.
110+
111+
Cost and token usage by model:
112+
113+
```sql
114+
SELECT
115+
response_model,
116+
gen_ai_system AS provider,
117+
count() AS calls,
118+
sum(total_tokens) AS tokens,
119+
round(sum(total_cost), 4) AS cost_usd
120+
FROM llm_metrics
121+
GROUP BY response_model, gen_ai_system
122+
ORDER BY cost_usd DESC
123+
LIMIT 20
124+
```
125+
126+
Spend per task:
127+
128+
```sql
129+
SELECT
130+
task_identifier,
131+
sum(input_tokens) AS input_tokens,
132+
sum(output_tokens) AS output_tokens,
133+
round(sum(total_cost), 4) AS cost_usd
134+
FROM llm_metrics
135+
GROUP BY task_identifier
136+
ORDER BY cost_usd DESC
137+
LIMIT 20
138+
```
139+
140+
Cost by prompt version, when calls are linked to an [AI Prompt](/ai/prompts):
141+
142+
```sql
143+
SELECT
144+
prompt_slug,
145+
prompt_version,
146+
count() AS calls,
147+
round(sum(total_cost), 4) AS cost_usd
148+
FROM llm_metrics
149+
WHERE prompt_slug != ''
150+
GROUP BY prompt_slug, prompt_version
151+
ORDER BY prompt_slug, prompt_version
152+
```
153+
154+
Set the time window with the query's [period filter](/observability/query#time-ranges) rather than in the SQL itself. Run these from the [Query dashboard](/observability/query#using-the-query-dashboard), the SDK with `query.execute()`, or the REST API. `llm_metrics` also exposes `ms_to_first_chunk` and `tokens_per_second` for latency and throughput, plus `finish_reason`, `request_model`, `cached_read_tokens`, `reasoning_tokens`, and per-direction `input_cost` / `output_cost` for finer breakdowns.
155+
156+
## Next steps
157+
158+
<CardGroup cols={2}>
159+
<Card title="Prompts" icon="message-lines" href="/ai/prompts">
160+
Version prompts as code and link generations to the exact prompt version that produced them.
161+
</Card>
162+
<Card title="Query (TRQL)" icon="magnifying-glass-chart" href="/observability/query">
163+
Write custom queries against your runs, metrics, and LLM usage.
164+
</Card>
165+
</CardGroup>

‎docs/docs.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,7 @@
107107
"group": "Features",
108108
"pages": [
109109
"ai/prompts",
110+
"ai/observability",
110111
"ai-chat/fast-starts",
111112
"ai-chat/compaction",
112113
"ai-chat/prompt-caching",

0 commit comments

Comments
 (0)