feat(tracing): add W3C distributed tracing and incremental handler batching

AGENTS.md

@@ -134,7 +134,13 @@ CI supports Python 3.11–3.14; root CI also spans Linux, macOS, and Windows.
 - Handler/decorator/OpenAI/ADK telemetry uses the internal logged-step path and converts completed steps to immutable
   OTel spans. The internal trace envelope is never exported as a span.
 - `start_splunk_ao_span()` is SDK-native OTel. `add_splunk_ao_span_processor()` and A2A instrument caller-owned OTel.
+  `configure_distributed_tracing()` is the combined automatic HTTP setup: it creates or accepts an application-owned
+  provider, registers Splunk AO export once per provider, configures supported upstream instrumentors, and returns the
+  provider for application shutdown. `instrument_distributed_tracing()` remains transport-only.
 - Never replace the process-global tracer provider. Register processors on the provided provider; respect ownership.
+- Keep SDK-owned authentication, validation, CRUD, routing-resolution, and other control-plane HTTP calls out of
+  application traces with scoped OTel HTTP suppression. Never use broad backend URL exclusions that can suppress user
+  traffic, and always restore instrumentation immediately after the SDK request.
 - Treat ended `ReadableSpan` objects as immutable. Normalize by copying at export, never by mutating private fields.
 - Completed spans enqueue immediately. `flush()` drains completed work without ending active work; `terminate()` drains,
   shuts down SDK-owned resources, and discards unfinished state. Caller-owned providers use `shutdown()`.

ARCHITECTURE.md

@@ -86,10 +86,40 @@ client/server wrapping and message-context propagation.
 The SDK may add processors, but it must never silently replace the global tracer provider. The application owns the
 provider and calls `shutdown()`; SDK-owned logger/export resources use their own termination path.
 
+### Distributed context and HTTP transports
+
+Supported automatic propagation reuses upstream OpenTelemetry instrumentors rather than custom SDK HTTP wrappers.
+`configure_distributed_tracing()` creates or accepts an OpenTelemetry SDK provider, registers Splunk AO export, and
+configures FastAPI/Starlette inbound instrumentation plus Requests, HTTPX sync/async, and aiohttp-client outbound
+instrumentation. It returns the application-owned provider for shutdown and never sets or replaces the global tracer
+provider. `instrument_distributed_tracing()` remains the transport-only entry point for applications that manage
+processor registration separately. Manual `get_tracing_headers()` and `TracingMiddleware` remain fallbacks for
+unsupported transports; supported automatic instrumentation does not require either one on individual requests.
+
+High-level setup is idempotent per provider for Splunk AO processor registration. Each process-wide client instrumentor
+and each server application must still have one instrumentation owner; do not combine the helper with direct upstream
+instrumentation of the same component.
+
+SDK-owned authentication, health-check, CRUD, routing-resolution, token-refresh, and streaming control-plane requests
+execute inside OTel HTTP-instrumentation suppression. This boundary is scoped to the SDK request and must restore the
+caller's context afterward; application traffic to the same host must remain instrumentable. Apply suppression at the
+shared configuration/API-client boundary, never by excluding deployment URLs globally.
+
+An explicit SDK session propagates as the standard W3C baggage member `gen_ai.conversation.id`. Export normalization
+may derive the local compatibility attribute `splunk_ao.session.id`, but that attribute is not propagated as a second
+baggage member. The SDK must not propagate authentication, deployment, Project, Agent Stream/log stream, experiment,
+endpoint, model, workflow, or agent identity in baggage.
+
 ## Span Lifecycle and Export Ownership
 
 `SpanSink` owns the SDK's private provider and `BatchSpanProcessor` for internal telemetry. A completed operation is
 enqueued immediately, allowing scheduled export without an explicit flush.
+Without an explicit internal `BatchConfig`, both SDK-owned and caller-owned processors defer to standard OTel
+`OTEL_BSP_*` environment configuration.
+
+Framework handlers register a stable operation identity at each start callback and enqueue that operation at its
+matching end callback, so a child can enter `BatchSpanProcessor` while its parent remains active. The deprecated
+`ingestion_hook` is the compatibility exception and retains whole-tree payload construction at trace completion.
 
 - `flush()` / `async_flush()` drain completed work and do not end an active trace.
 - `terminate()` drains completed work, shuts down SDK-owned telemetry resources, and releases unfinished state.
@@ -156,6 +186,9 @@ receiver failures, and expose bounded acknowledgement health. Do not log secrets
 
 These handlers adapt framework events into the internal Path 1 model. Keep optional dependencies lazy and imports free
 of surprising side effects. CrewAI is excluded on Python 3.14; code and tests must support the unavailable path.
+With `start_new_trace=True`, a handler owns and concludes its local envelope and root while still inheriting an active
+W3C parent. With `start_new_trace=False`, it attaches below caller-owned active state, emits only its subtree, and leaves
+the caller open.
 
 ### OpenAI wrapper
 

CHANGELOG.md

@@ -7,6 +7,50 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Added the `distributed-tracing` extra and high-level
+  `configure_distributed_tracing()` setup for Splunk AO export plus supported
+  upstream FastAPI/Starlette, Requests, HTTPX, and aiohttp-client instrumentation.
+  The lower-level `instrument_distributed_tracing()` transport helper remains
+  available for applications that configure their provider separately.
+- Explicit SDK sessions now propagate across supported services as standard
+  W3C `gen_ai.conversation.id` baggage. SDK routing, authentication, and
+  application identity are not propagated.
+- Added module-level `get_tracing_headers()` and
+  `extract_tracing_context()` helpers for standard W3C `traceparent` and
+  `tracestate` propagation. `TracingMiddleware` now extracts and scopes that
+  OpenTelemetry context for Starlette/FastAPI requests.
+- SDK-owned authentication, validation, and CRUD HTTP requests are suppressed
+  from automatic application HTTP instrumentation, preventing control-plane
+  calls from appearing as unrelated application traces.
+
+### Changed
+
+- The temporarily retained `mode="batch"` and `mode="distributed"` values now
+  use identical scheduled OTLP batch export and W3C propagation. Concluding an
+  operation ends and queues it; a per-operation `flush()` is not required.
+- Callback handlers keep a live real root during framework execution so
+  outbound W3C context uses the same identity and visible hierarchy that is
+  exported at commit.
+- Normal LangChain, CrewAI, Google ADK, and OpenAI Agents callbacks now enqueue
+  each completed operation into the existing `BatchSpanProcessor` at that
+  operation's end callback. The deprecated `ingestion_hook` retains its
+  whole-tree compatibility behavior.
+- Default logger-owned batching now honors standard OpenTelemetry
+  `OTEL_BSP_*` configuration, matching caller-owned OTel paths. Explicit
+  internal `BatchConfig` values remain authoritative when supplied.
+
+### Removed
+
+- **Breaking:** Removed `trace_id=` and `span_id=` from `SplunkAOLogger`, the
+  logger-level proprietary `get_tracing_headers()` method, and custom
+  `Splunk-AO-Trace-ID` / `Splunk-AO-Parent-ID` continuation. Use the new
+  module-level W3C helpers instead.
+- Removed the obsolete distributed-only REST streaming worker and task queue;
+  normal telemetry in both retained modes uses the existing
+  `BatchSpanProcessor` path.
+
 ## [0.2.1] - 2026-08-07
 
 ### Fixed

README.md

@@ -32,6 +32,7 @@ Install the optional integration dependencies used by your application:
 ```shell
 pip install "splunk-ao[openai]"
 pip install "splunk-ao[langchain]" langchain-openai
+pip install "splunk-ao[distributed-tracing]"
 ```
 
 Other available extras include `crewai`, `middleware`, and `all`.
@@ -347,7 +348,8 @@ tracer_provider.shutdown()
 `add_splunk_ao_span_processor()` reads the same deployment and routing
 configuration described in [Setup](#setup). It registers a
 `SplunkAOSpanProcessor`, which uses a standard OpenTelemetry
-`BatchSpanProcessor`.
+`BatchSpanProcessor`. Default logger-owned and caller-owned processors honor
+the standard `OTEL_BSP_*` batch processor environment variables.
 
 If your application needs another processor or exporter, register it through
 the standard OpenTelemetry extension points. For example, to send the same
@@ -364,6 +366,49 @@ The application owns a `TracerProvider` that it constructs, so it must call
 `tracer_provider.shutdown()` during teardown. A `SplunkAOLogger` created
 elsewhere owns a separate provider and should be terminated separately.
 
+##### Automatic HTTP distributed tracing
+
+The `distributed-tracing` extra provides a supported one-time setup for
+upstream OpenTelemetry FastAPI/Starlette, Requests, HTTPX sync/async, and
+aiohttp-client instrumentation. The high-level helper creates an application-owned
+provider, registers Splunk AO export, and configures those transports together:
+
+```python
+from fastapi import FastAPI
+from splunk_ao import configure_distributed_tracing
+
+app = FastAPI()
+provider = configure_distributed_tracing(app=app)
+```
+
+The application owns the returned `provider` and calls `provider.shutdown()`
+during process teardown. The helper never replaces the process-global provider.
+Pass an existing OpenTelemetry SDK `TracerProvider` through the
+`tracer_provider` argument when other OpenTelemetry/OpenInference agent or model
+instrumentations should use that same provider and join the trace.
+
+For advanced composition, `add_splunk_ao_span_processor()` remains the
+export-only API and `instrument_distributed_tracing()` remains the supported
+transport-only API. Do not also directly instrument a client or application
+component that is owned by the high-level helper; each component should have one
+instrumentation owner.
+
+After this startup call, supported inbound requests extract W3C context and
+supported outbound clients inject it automatically. Application code does not
+call `get_tracing_headers()` for each request and does not install
+`TracingMiddleware`. Those APIs remain supported for transports and frameworks
+outside the automatic support matrix.
+
+SDK-owned authentication, health-check, Project, Agent Stream, and other
+control-plane HTTP requests are scoped out of automatic client instrumentation.
+They therefore do not appear as application traces, while HTTP requests made by
+the application remain instrumented normally.
+
+An explicit SDK session propagates as the standard
+`gen_ai.conversation.id` W3C baggage member. Project, Agent Stream,
+experiment, deployment, authentication, model, workflow, and agent identity
+remain local and are never added to baggage by the SDK.
+
 #### Export diagnostics
 
 The SDK does not change your application's global logging configuration.
@@ -416,6 +461,19 @@ not close a trace that was started by the caller. In short-lived jobs, ensure
 the owning logger or provider is terminated or shut down after framework work
 finishes.
 
+For normal OTLP export, LangChain, CrewAI, Google ADK, and OpenAI Agents
+handlers enqueue each operation into the existing `BatchSpanProcessor` when
+that operation's end callback runs. A child can therefore enter a scheduled
+or size-based export batch while its parent remains active. This is immediate
+enqueue, not one network request per span, and no per-trace `flush()` is
+required. The deprecated `ingestion_hook` compatibility path still constructs
+one mutable whole-tree payload at trace completion.
+
+`start_new_trace` controls ownership, not whether distributed tracing works.
+Keep its default `True` for a standalone handler. Set it to `False` only when
+the handler is intentionally placed beneath caller-owned active logger state;
+the handler emits its own subtree and does not conclude the caller.
+
 #### Datasets
 
 Create a dataset:

examples/logging-samples/DT_2.0/distributed-tracing-auto/.env.example

@@ -0,0 +1,23 @@
+# Select exactly one deployment block. Do not mix o11y and standalone variables.
+# O11y Cloud
+SPLUNK_AO_REALM="your-o11y-realm"
+SPLUNK_AO_O11Y_TOKEN="your-o11y-ingest-or-ingest+api-token"
+# SPLUNK_AO_O11Y_API_TOKEN="your-o11y-api-token"
+
+# Standalone
+SPLUNK_AO_API_KEY="your-splunk-ao-api-key"
+SPLUNK_AO_CONSOLE_URL="https://app.galileo.ai"
+
+# Configure project and agentstream
+SPLUNK_AO_PROJECT="your-splunk-ao-project"
+SPLUNK_AO_AGENT_STREAM="distributed-tracing-example"
+
+# OpenAI configuration
+OPENAI_API_KEY="your-openai-api-key"
+OPENAI_MODEL=gpt-5-nano
+
+OTEL_SERVICE_NAME=distributed-tracing-manual
+
+# Optional delay to see partial traces.
+OTEL_BSP_SCHEDULE_DELAY="1000"
+BATCH_DEMO_SLEEP_SECONDS="20"

examples/logging-samples/DT_2.0/distributed-tracing-auto/README.md

@@ -0,0 +1,39 @@
+## Configure
+
+Copy `.env.example` to `.env` and set the credentials and project and agentstream.
+
+```bash
+python -m venv venv
+source venv/bin/activate
+pip install -r requirements.txt
+cp .env.example .env
+```
+
+Start the retrieval service:
+```bash
+uvicorn retrieval_service:app --port 8000
+```
+
+Then run the orchestrator:
+```bash
+python main_run.py
+```
+
+### Observe batching before the root ends
+
+To demonstrate incremental export, add these values to an existing `.env`:
+
+```dotenv
+OTEL_BSP_SCHEDULE_DELAY="1000"
+BATCH_DEMO_SLEEP_SECONDS="20"
+```
+
+`OTEL_BSP_SCHEDULE_DELAY` is in milliseconds and is read when each batch span
+processor is created. After the distributed retrieval call returns, the example
+prints a batching-demo message and keeps the orchestrator root active for 20
+seconds.
+
+Set`BATCH_DEMO_SLEEP_SECONDS="0"` to disable the pause.
+
+The local tree remains the same as the baseline. Completed children enter the
+batch processor at their own end callbacks, so no per-trace flush is needed.

examples/logging-samples/DT_2.0/distributed-tracing-auto/main_run.py

@@ -0,0 +1,90 @@
+"""Orchestrator using automatic W3C HTTP client instrumentation."""
+
+import asyncio
+import os
+from uuid import uuid4
+
+import httpx
+from dotenv import load_dotenv
+
+from splunk_ao import configure_distributed_tracing, log, openai, splunk_ao_context
+
+load_dotenv()
+
+provider = configure_distributed_tracing()
+
+openai_client = openai.OpenAI()
+OPENAI_MODEL = os.environ["OPENAI_MODEL"]
+RETRIEVAL_SERVICE_URL = "http://localhost:8000"
+BATCH_DEMO_SLEEP_SECONDS = float(os.getenv("BATCH_DEMO_SLEEP_SECONDS", "0"))
+
+
+@log
+async def orchestrator_agent(question: str) -> str:
+    """Run one independently traced RAG request."""
+    analysis = analyze_question(question)
+
+    # HTTPX creates the client span and injects the active W3C context.
+    async with httpx.AsyncClient(base_url=RETRIEVAL_SERVICE_URL, timeout=100.0) as client:
+        try:
+            response = await client.post("/retrieve", json={"query": question})
+            response.raise_for_status()
+            retrieved_docs = response.json()["results"]
+        except httpx.HTTPError:
+            retrieved_docs = []
+
+    if BATCH_DEMO_SLEEP_SECONDS > 0:
+        print(
+            "Batching demo: analysis and distributed retrieval spans have ended; "
+            f"the orchestrator root remains active for {BATCH_DEMO_SLEEP_SECONDS:g} seconds."
+        )
+        await asyncio.sleep(BATCH_DEMO_SLEEP_SECONDS)
+
+    context = format_context(analysis, retrieved_docs)
+    response = openai_client.chat.completions.create(
+        messages=[
+            {
+                "role": "system",
+                "content": (
+                    "Answer the user's question using only the provided context. "
+                    f"If the context is insufficient, say so.\n\nContext:\n{context}"
+                ),
+            },
+            {"role": "user", "content": question},
+        ],
+        model=OPENAI_MODEL,
+    )
+    return response.choices[0].message.content or ""
+
+
+@log
+def analyze_question(question: str) -> dict[str, object]:
+    question_lower = question.lower()
+    return {
+        "needs_company_info": any(word in question_lower for word in ("company", "work", "employer")),
+        "needs_location_info": any(word in question_lower for word in ("location", "where", "city", "live")),
+        "question_type": "factual",
+    }
+
+
+@log
+def format_context(analysis: dict[str, object], documents: list[str]) -> str:
+    rendered = [f"Analysis: {analysis}", "", "Retrieved Documents:"]
+    rendered.extend(f"{index}. {document}" for index, document in enumerate(documents, 1))
+    return "\n".join(rendered)
+
+
+async def main() -> None:
+    # One app-owned session is propagated as gen_ai.conversation.id baggage.
+    # set_session() is local-only; unlike start_session(), it performs no CRUD request.
+    splunk_ao_context.set_session(str(uuid4()))
+    for question in ("What did Galileo Galilei research?", "Where did Galileo Galilei work?"):
+        answer = await orchestrator_agent(question)
+        print(f"Question: {question}\nAnswer: {answer}\n")
+
+
+if __name__ == "__main__":
+    try:
+        asyncio.run(main())
+    finally:
+        provider.shutdown()

examples/logging-samples/DT_2.0/distributed-tracing-auto/requirements.txt

@@ -0,0 +1,6 @@
+python-dotenv==1.2.2
+pydantic==2.11.9
+httpx==0.28.1
+fastapi==0.117.1
+uvicorn==0.37.0
+-e ../../../..[openai,distributed-tracing]