feat(appkit): complete MLflow UC tracing across agent lifecycle

[adamgurary]

Summary

This continues and supersedes the tracing architecture introduced in #477. It keeps one AppKit OpenTelemetry provider and attaches MLflow/Unity Catalog processing to that provider, instead of creating a second mlflow-tracing provider. The existing run-linking behavior is preserved through the semantic mlflow.sourceRun root attribute.

Companion rollout across executable templates: databricks/app-templates#260

The change makes MLflow tracing a first-class AppKit agent lifecycle contract:

• creates semantic AGENT, CHAT_MODEL, TOOL, RETRIEVER, MEMORY, CHAIN, PARSER, and nested-agent spans across agent routes, adapters, tools, AI Search, MCP, model serving, caches, parsers, and Supervisor integrations;
• captures bounded inputs, outputs, errors, model metadata, token/cache usage, estimated cost when available, latency, time to first token, streaming duration, and request/session/user/thread/route identity;
• propagates W3C trace context across local and remote boundaries and exposes stable MLflow trace ID aliases;
• exports the same span tree to Unity Catalog through authenticated OTLP plus MLflow V4 trace registration, with setup/provisioning support and immutable resource binding validation;
• adds conformance, lifecycle, connector, parser, security-limit, and end-to-end template coverage so new agent execution paths cannot silently omit tracing.

Architecture relative to #477

#477 introduced plugin-local MLflow tracing backed by a separate mlflow-tracing provider. This PR folds that intent into AppKit's existing telemetry system: one provider owns the span lifecycle, while the UC processor/exporter handles MLflow persistence. This avoids split or orphaned trace trees and lets every AppKit agent surface share the same propagation, serialization, usage, cost, and error semantics.

Verification

• Focused agent/UC tracing suites: 44 passed
• Agent adapter/parser suites: 53 passed
• Full test suite: 4,452 passed, 2 skipped
• Typecheck: passed
• pnpm check: passed (89 existing warnings)
• Package build: passed
• Documentation build: passed
• All 29 contribution commits include DCO sign-off
• OSS scrub: no internal URLs, workspace identifiers, warehouse identifiers, experiment identifiers, or credentials; remaining IDs are synthetic fixtures

Live-environment limitations

• Managed Supervisor still needs live validation that it continues the supplied W3C context or returns a persistently linkable remote trace identity. The implementation propagates context and records returned identity, but this PR does not claim that managed-service behavior has been proven.
• No live Databricks credential/profile was available for this verification run.
• The Apps Unity Catalog experiment resource used during earlier investigation returned HTTP 403.
• Therefore, this PR does not claim live Unity Catalog persistence was validated; local contract, exporter, registration, propagation, and conformance behavior are covered by tests.

[adamgurary]

@MarioCadenas GitHub has queued PR Metadata Verification, Bundle Size, and CI as action_required for this first fork contribution. Could you approve those workflow runs when you review? The companion all-agent-template rollout is databricks/app-templates#260.

[adamgurary]

Ran the repo-local /review-pr origin/main command repeatedly against the full committed branch and addressed the substantiated findings.

Review fixes now included:

• corrected the MLflow V4 trace-info wire contract and UC spans-table field
• bounded workspace resolution, authentication, trace-info, OTLP retries, pending traces, retained spans, completed-export concurrency, and export queue depth
• added fresh per-request auth, non-retryable OTLP 4xx handling, response-body release, and memoized shutdown
• hardened cyclic/shared/BigInt/Date/toJSON input and output serialization
• made destructive-tool HITL preflight cover nested sub-agents
• switched agent serving endpoint discovery to DATABRICKS_AGENT_SERVING_ENDPOINT_NAME with the legacy fallback
• made internal stream IDs server-owned and concurrency reservation atomic
• preserved bounded, secret-scrubbed AI Search retriever diagnostics
• made setup preview read-only, prevented partial setup writes, and added an actionable missing-uv error
• honored an explicit mlflowUc: false opt-out

One review recommendation was intentionally not adopted: agents still enable UC tracing by default and missing or invalid UC configuration still fails startup. That is the approved fail-fast requirement for this contribution, so tracing cannot silently disappear from an agent deployment.

Verification:

• 4,477 tests passed; 2 skipped
• full workspace typecheck passed
• pnpm check passed
• package build, docs build, and publint/attw passed
• MLflow 3.14 provisioner suite: 10/10 passed
• DCO trailers and secret scrub verified