diff --git a/content/docs/ingest-data/ai-agents/agent-frameworks/meta.json b/content/docs/ingest-data/ai-agents/agent-frameworks/meta.json index f653242..cf4370c 100644 --- a/content/docs/ingest-data/ai-agents/agent-frameworks/meta.json +++ b/content/docs/ingest-data/ai-agents/agent-frameworks/meta.json @@ -5,6 +5,7 @@ "[CrewAI](/ingest-data/ai-agents/crewai)", "[n8n](/ingest-data/ai-agents/n8n)", "[Mastra](/ingest-data/ai-agents/mastra)", + "[OpenLIT](/ingest-data/ai-agents/openlit)", "[LangChain](/ingest-data/ai-agents/langchain)", "[LlamaIndex](/ingest-data/ai-agents/llamaindex)", "[AutoGen](/ingest-data/ai-agents/autogen)", diff --git a/content/docs/ingest-data/ai-agents/meta.json b/content/docs/ingest-data/ai-agents/meta.json index 5134103..81c1731 100644 --- a/content/docs/ingest-data/ai-agents/meta.json +++ b/content/docs/ingest-data/ai-agents/meta.json @@ -6,4 +6,4 @@ "llms", "gateways" ] -} +} \ No newline at end of file diff --git a/content/docs/ingest-data/ai-agents/openlit.mdx b/content/docs/ingest-data/ai-agents/openlit.mdx new file mode 100644 index 0000000..6269697 --- /dev/null +++ b/content/docs/ingest-data/ai-agents/openlit.mdx @@ -0,0 +1,284 @@ +--- +title: OpenLIT Agent Observability +description: Send agent, workflow, tool, and model telemetry from OpenLIT to Parseable +--- + +import { Step, Steps } from 'fumadocs-ui/components/steps'; + +[OpenLIT](https://openlit.io/) is an OpenTelemetry-native observability SDK for +LLM applications and AI agents. It can instrument supported agent frameworks, +model providers, and tools so one trace shows the complete path from an agent +invocation to its workflow steps, tool executions, and model calls. + +This integration is framework-neutral. Use the same Parseable and OpenTelemetry +Collector configuration with CrewAI, OpenAI Agents SDK, LangChain or LangGraph, +Agno, LlamaIndex, and other frameworks supported by OpenLIT. + +```text +Agent application + | + | OpenLIT instrumentation + v +Agent invocation -> workflow -> tool calls -> model calls + | + | OTLP/HTTP + v +OpenTelemetry Collector + | + +--> openlit-agent-traces Agent Observability and trace dataset + +--> openlit-agent-metrics Metrics dataset +``` + +## Prerequisites + +- A running Parseable instance +- A Parseable API key with dataset creation and ingest access +- Python 3.9 or newer +- An OpenTelemetry Collector Contrib binary or container +- A supported agent framework and its model-provider API key + +## Set up OpenLIT Agent Observability + + + + +### Create the Parseable datasets + +Set the connection values: + +```bash +export PARSEABLE_URL="https://:8000" +export PARSEABLE_API_KEY="" +export OPENLIT_TRACE_STREAM="openlit-agent-traces" +export OPENLIT_METRIC_STREAM="openlit-agent-metrics" +``` + +Create the trace dataset and tag it for Agent Observability: + +```bash +curl -X PUT "$PARSEABLE_URL/api/v1/logstream/$OPENLIT_TRACE_STREAM" \ + -H "X-API-Key: ${PARSEABLE_API_KEY}" \ + -H "X-P-Log-Source: otel-traces" \ + -H "X-P-Telemetry-Type: traces" \ + -H "X-P-Dataset-Tag: agent-observability" +``` + +Create the metrics dataset: + +```bash +curl -X PUT "$PARSEABLE_URL/api/v1/logstream/$OPENLIT_METRIC_STREAM" \ + -H "X-API-Key: ${PARSEABLE_API_KEY}" \ + -H "X-P-Log-Source: otel-metrics" \ + -H "X-P-Telemetry-Type: metrics" +``` + +`X-P-Dataset-Tag: agent-observability` makes the trace dataset available from +Parseable's Agents page. Create the dataset explicitly because an automatically +created OTLP trace dataset does not receive this tag. + + + + +### Configure the OpenTelemetry Collector + +Create `otel-collector-config.yaml`: + +```yaml +receivers: + otlp: + protocols: + http: + endpoint: 0.0.0.0:4318 + +processors: + batch: + timeout: 5s + +exporters: + otlphttp/parseable_traces: + endpoint: "${env:PARSEABLE_URL}" + encoding: json + headers: + X-API-Key: "${env:PARSEABLE_API_KEY}" + X-P-Stream: "${env:OPENLIT_TRACE_STREAM}" + X-P-Log-Source: otel-traces + + otlphttp/parseable_metrics: + endpoint: "${env:PARSEABLE_URL}" + encoding: json + headers: + X-API-Key: "${env:PARSEABLE_API_KEY}" + X-P-Stream: "${env:OPENLIT_METRIC_STREAM}" + X-P-Log-Source: otel-metrics + +service: + pipelines: + traces: + receivers: [otlp] + processors: [batch] + exporters: [otlphttp/parseable_traces] + + metrics: + receivers: [otlp] + processors: [batch] + exporters: [otlphttp/parseable_metrics] +``` + +Start the Collector in the shell containing the exported variables: + +```bash +otelcol-contrib --config otel-collector-config.yaml +``` + +Use the Parseable base URL. The `otlphttp` exporter appends `/v1/traces` and +`/v1/metrics` for the corresponding pipelines. + + + + +### Instrument the agent application + +Install OpenLIT and the packages required by your chosen agent framework: + +```bash +pip install openlit +``` + +Initialize OpenLIT before importing the framework or provider SDK: + +```python +import openlit + +openlit.init( + otlp_endpoint="http://localhost:4318", + service_name="support-agent", + environment="production", + capture_message_content=False, +) + +# Import and run your supported agent framework only after initialization. +``` + +No Parseable-specific instrumentation belongs in the agent implementation. The +Collector determines which trace and metric datasets receive the OTLP signals. + +Choose the OpenLIT integration matching your framework: + +- [OpenAI Agents SDK](https://docs.openlit.io/latest/sdk/integrations/openai-agents) +- [CrewAI](https://docs.openlit.io/latest/sdk/integrations/crewai) +- [LangChain and LangGraph](https://docs.openlit.io/latest/sdk/integrations/langchain) +- [All OpenLIT integrations](https://docs.openlit.io/latest/sdk/integrations/overview) + +The application must execute an agent framework operation. A direct model SDK +call produces model-operation spans such as `chat`, but it does not create an +agent invocation, workflow, or tool-execution hierarchy. + + + + +### Run and verify an agent + +Run an agent workflow that makes at least one model call and, when possible, +executes a tool. Allow the SDK and Collector batch processors to flush. + +In Parseable, verify: + +1. `openlit-agent-traces` appears in both Traces and Agents. +2. One trace contains the complete agent run and its child spans. +3. Agent or workflow operations appear alongside model operations such as `chat`. +4. Tool spans represent actual executions, not only tool-call requests returned by a model. +5. All related spans share a trace ID and have coherent parent span IDs. +6. `openlit-agent-metrics` appears in Metrics. + +For a short-lived smoke test, reduce the metric export interval: + +```bash +export OTEL_METRIC_EXPORT_INTERVAL=1000 +python agent.py +``` + + + + +## What you get in Parseable + +Exact span names and attributes depend on the framework, provider, and OpenLIT +version. A complete agent trace can contain: + +| Telemetry | What it represents | +| --- | --- | +| Agent invocation | Top-level agent execution | +| Workflow or task | Chain, graph, crew, task, or handoff operation | +| Tool execution | Tool name, arguments, result, duration, and status | +| Model operation | Provider request, model, tokens, cost, latency, and response status | +| Error event | Framework, tool, or provider exception with trace context | + +Common fields include: + +| Field | Meaning | +| --- | --- | +| `service.name` | Service configured in `openlit.init()` | +| `deployment.environment` | Deployment environment | +| `gen_ai.operation.name` | Operation such as `invoke_agent`, `invoke_workflow`, or `chat` | +| `gen_ai.agent.name` | Agent name when supplied by the framework | +| `gen_ai.workflow.name` | Workflow name when supplied by the framework | +| `gen_ai.tool.name` | Tool name on supported tool spans or model tool-call data | +| `gen_ai.request.model` | Requested model | +| `gen_ai.usage.input_tokens` | Input token count | +| `gen_ai.usage.output_tokens` | Output token count | +| `gen_ai.usage.cost` | Computed model-request cost | +| `span_trace_id` | Identifier joining all spans in one run | +| `span_parent_span_id` | Parent relationship used by the waterfall | +| `span_status_code` | Operation status | + +OpenLIT also emits metrics such as `gen_ai.client.operation.duration`, +`gen_ai.client.token.usage`, `gen_ai.usage.cost`, +`gen_ai.server.time_to_first_token`, and +`gen_ai.server.time_per_output_token`. + +## Query agent telemetry + +Agent and workflow operations: + +```sql +SELECT + "gen_ai.operation.name" AS operation, + COUNT(*) AS spans, + COUNT(DISTINCT "span_trace_id") AS runs +FROM "openlit-agent-traces" +WHERE p_timestamp > NOW() - INTERVAL '24 hours' +GROUP BY operation +ORDER BY spans DESC; +``` + +Model usage caused by agent runs: + +```sql +SELECT + "gen_ai.request.model" AS model, + COUNT(*) AS calls, + SUM(CAST("gen_ai.usage.input_tokens" AS BIGINT)) AS input_tokens, + SUM(CAST("gen_ai.usage.output_tokens" AS BIGINT)) AS output_tokens +FROM "openlit-agent-traces" +WHERE "gen_ai.operation.name" = 'chat' + AND p_timestamp > NOW() - INTERVAL '24 hours' +GROUP BY model +ORDER BY calls DESC; +``` + +## Troubleshoot + +- **Dataset appears in Traces but not Agents:** The dataset was probably auto-created without the Agent Observability tag. Recreate an empty dataset with `X-P-Log-Source: otel-traces`, `X-P-Telemetry-Type: traces`, and `X-P-Dataset-Tag: agent-observability` before ingesting data. +- **Traces contain only `chat` operations:** OpenLIT is observing direct model calls, not an agent-framework execution. Confirm that the installed framework is supported, initialize OpenLIT before importing it, and execute the framework's agent runner rather than only its underlying model client. +- **Tool calls appear without tool execution spans:** A model can request a tool without the application executing it. Confirm that the agent framework dispatches the tool and that OpenLIT supports instrumenting that framework's tool runtime. +- **No traces appear:** Check Collector logs, confirm the application exports OTLP/HTTP to port `4318`, and verify the Parseable exporter includes `X-P-Stream` and `X-P-Log-Source: otel-traces`. +- **Traces appear but metrics are empty:** Metrics use a periodic exporter. Keep the application alive until the first export or temporarily set `OTEL_METRIC_EXPORT_INTERVAL=1000`. +- **Prompts or responses appear in traces:** Set `capture_message_content=False`. This affects only newly generated telemetry; review or remove older records separately. + +## Related documentation + +- [OpenLIT SDK](https://docs.openlit.io/latest/sdk/quickstart-ai-observability) +- [Parseable Agent Observability](/docs/user-guide/agent-observability) +- [Parseable OpenTelemetry traces](/docs/ingest-data/otel/traces) +- [Parseable OpenTelemetry metrics](/docs/ingest-data/otel/metrics) +- [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/)