feat(evaluations): preserve the tool trajectory for judges

examples/graph_streaming.py

@@ -6,7 +6,7 @@
 Like ``examples/streaming.py``, the generator is built inside ``conversation_id`` and iterated
 *outside* it. That is the shape a server produces when it hands a stream to a transport, and it
 is what exercises call-time binding: an async generator body does not run until the first
-``__anext__``, so both the conversation id and the ``ld.ai.graph`` span's OTel parent have to be
+``__anext__``, so both the conversation id and the ``launchdarkly.graph`` span's OTel parent have to be
 captured when ``stream()`` is called, not when iteration starts.
 
 Writes no JSON output file, same as the single-config streaming example.

packages/ai/README.md

@@ -73,7 +73,7 @@ result = await evals.run(
 )
 ```
 
-`LD_API_TOKEN` is required. Configure `LD_SDK_KEY` — or initialize your own client with `init_client(client=...)` — to emit one `$ld:ai:offline-evals:generation` event per generated row, plus one `$ld:ai:offline-evals:criterion` event per `(row, criterion)` when `criteria` are supplied, through the standard SDK event transport. The SDK reports scores; LaunchDarkly rules on them at ingest. A judge served by a different provider than `generation` needs a handler for it in `judge_handlers`. Use `LD_API_BASE_URI` for staging or local management API traffic; it is separate from the SDK delivery setting `LD_BASE_URI`. Evaluation-run links use the explicit `ui_base_uri` option or `LD_UI_BASE_URI`, defaulting to `https://app.launchdarkly.com`; set it when the project is not in production, or a run created elsewhere still links to the production app. See the [core evaluations guide](../client/README.md#run-an-evaluation-from-code).
+`LD_API_TOKEN` is required. Configure `LD_SDK_KEY` — or initialize your own client with `init_client(client=...)` — to emit one `$ld:ai:offline-evals:generation` event per generated row, plus one `$ld:ai:offline-evals:criterion` event per `(row, criterion)` when `criteria` are supplied, through the standard SDK event transport. The SDK reports scores; LaunchDarkly rules on them at ingest. A judge served by a different provider than `generation` needs a handler for it in `judge_handlers`. Each row's tool calls are recorded during generation and rendered into the judge's `{{message_history}}`, between the row input and the generated output, so a rubric can grade the tool trajectory as well as the final answer. Use `LD_API_BASE_URI` for staging or local management API traffic; it is separate from the SDK delivery setting `LD_BASE_URI`. Evaluation-run links use the explicit `ui_base_uri` option or `LD_UI_BASE_URI`, defaulting to `https://app.launchdarkly.com`; set it when the project is not in production, or a run created elsewhere still links to the production app. See the [core evaluations guide](../client/README.md#run-an-evaluation-from-code).
 
 ---
 

packages/client/README.md

@@ -116,6 +116,51 @@ result = await init_evaluations().run(
 
 **The SDK reports scores and never rules on them.** LaunchDarkly derives each row's verdict at ingest by comparing the score against the criterion's stored threshold and success direction, so pass/fail policy is one server-side implementation that applies to every SDK version and to runs already recorded. A judge's direction lives on its AI Config and is injected server-side, keeping the one input a verdict turns on server-attested; a `Scorer` has no LaunchDarkly-side config to read, so it declares its own `success_direction` (default `"higher_is_better"` — set `"lower_is_better"` for a scorer that counts something unwanted, like a regex hit count).
 
+#### Judge the tool trajectory
+
+A judge is shown the tool calls the row made on the way to its output, so a rubric can grade *how* the agent answered and not only *what* it answered — whether it called the right tool, in the right order, with the right arguments, and how it handled a tool that failed.
+
+The harness records this itself: it wraps your tool implementations once per row before handing them to the handler, so every handler package is covered without changes and your tools still return and raise exactly what they did before.
+
+**This is not specific to offline evaluations.** Online judges — both the inline ones sampled by `config().invoke()` and the deferred ones you run from a `JudgeTask` on a background thread — are shown the same trajectory, built by the same function. See [Judges see one conversation](#judges-see-one-conversation).
+
+The trajectory is rendered into **`{{message_history}}`** — the row input, then the trajectory, then the generated output, then the formatting instructions, in that order. There is no separate trajectory variable: `message_history` is already the transcript variable every judge reads, and judges built from the AI Library's default templates reference it, so a trajectory rubric can be written against an existing judge template with no new placeholder.
+
+```
+Tools available: lookup_order, issue_refund
+Tool calls made while producing the response, in order:
+1. lookup_order
+   arguments: {"id":"A1"}
+   result: order A1 shipped 2026-08-02
+2. issue_refund
+   arguments: {"id":"A1","amount":19.99}
+   error: refund window closed
+```
+
+A row with tools that called none of them says so explicitly, which is the finding a tool-selection rubric most needs. A run with no tools adds no block at all, so judges written before trajectories existed read exactly the history they read before.
+
+#### Judges see one conversation
+
+All three judge paths build `{{message_history}}` through a single function, `judge_scoring.build_message_history`:
+
+| Path | Entry point |
+| --- | --- |
+| Online, inline | `config().invoke()` → `run_judges` |
+| Online, deferred | `config(skip_judges=True).invoke()` → `run_judge(task, handlers)` on your own thread |
+| Offline | `init_evaluations().run(criteria=[Judge(...)])` |
+
+Each one is the input, then the tool trajectory, then the output, then the `{score, reasoning}` format block, with empty parts skipped. A judge therefore grades the same conversation wherever it runs, which is what makes a rubric portable between a production sample and a dataset replay.
+
+They did not always agree, and that is why this is a single function now: each path used to join its own history. The offline one carried the row input, the inline one carried the user input, and the deferred one carried **neither** — so a deferred judge graded a response with no request beside it. `JudgeTask` gained `user_input` and `trajectory` to close that.
+
+For the deferred path those two fields travel on the task, which stays picklable — the trajectory crosses as the rendered string, not the structured record.
+
+A **graph-level** judge (`graph_judge`) gets no trajectory: it grades a final answer produced across several nodes, and splicing their trajectories together would describe a conversation that never happened. Per-node judges inside a graph do get their own node's.
+
+Two limits keep a trajectory from spending the judge's context window: at most 50 recorded calls per row and 2000 characters per rendered argument bag or result, with anything beyond either reported as a count or marked truncated. Calls past the limit still execute — truncation drops the record, never the work. A `NativeTool` runs inside the provider, so no local wrapper sees it; such a tool is left out of the trajectory and out of the "Tools available" line, since naming a tool whose use cannot be shown would invite a judge to conclude the model ignored it.
+
+A tool result is now judge-prompt input. It stays literal for the same reason the generated output does: the judge config is handed to the handler unrendered and the handler makes exactly one template pass, so a `{{...}}` sequence coming back from a tool is never expanded into the judge prompt.
+
 **Judges are independent AI Configs, so handlers are routed per judge.** A judge may resolve to a different provider or mode than `generation`, and a handler built for one provider cannot execute another's config. `handler` runs a judge when it provides for that judge's provider; pass handlers for any other providers in `judge_handlers`. Selection prefers a handler naming the judge's provider outright over a wildcard multi-provider adapter, and an agent-mode handler can serve a messages-mode judge with its messages collapsed into one instructions block. A plain callable that declares no `provides_for` routes itself, exactly as it already does for the generation config.
 
 Judges are resolved through flag delivery, and handlers are matched to them, **before** any evaluation records are created — a missing judge or one no handler covers fails the run up front rather than after the generation spend. After that point a criterion failure never aborts the run: an unparseable judge response, an out-of-range score, a raising handler or scorer, and a row whose generation errored each become a per-criterion `ERROR` event with a cause code (`invalid_judge_output`, `invalid_score`, `handler_raised`, `scorer_raised`, `generation_incomplete`) and a top-level `errorMessage`. Event *delivery* is different: the backend needs one result per `(row, criterion)` to finish row accounting, so if tracking a criterion event fails, every remaining result is still attempted and flushed and then `run()` raises — rather than polling to its timeout with the cause hidden.

packages/client/src/launchdarkly_ai_server/client.py

@@ -123,6 +123,8 @@ async def invoke(
                 handlers=resolved_handler_list,
                 llm_response=llm_str,
                 base_track_data=track_data,
+                user_input=user_input,
+                trajectory=result.get("trajectory", ""),
             )
             return ProviderResponse(
                 response=parsed_response,
@@ -137,6 +139,7 @@ async def invoke(
             handler=handler,
             handlers=resolved_handler_list,
             user_input=user_input,
+            trajectory=result.get("trajectory", ""),
             llm_response=llm_str,
             base_track_data=track_data,
             tool_handlers=resolved_tools,
@@ -215,6 +218,7 @@ async def _stream_events(
                     handler=handler,
                     handlers=resolved_handler_list,
                     user_input=user_input,
+                    trajectory=done_event.get("trajectory", ""),
                     llm_response=done_event.get("response", ""),
                     base_track_data=track_data,
                     tool_handlers=resolved_tools,

packages/client/src/launchdarkly_ai_server/evaluations/runner.py

@@ -14,10 +14,16 @@
 
 from ..judge_scoring import (
     FORMATTING_INSTRUCTIONS,
+    build_message_history,
     numeric_score,
     parse_judge_response,
 )
 from ..lifecycle import extract_variation
+from ..trajectory import (
+    TrajectoryRecorder,
+    render_row_trajectory,
+    row_fields,
+)
 from ..types import NativeTool
 from ..utils import (
     collapse_messages_to_instructions,
@@ -624,11 +630,15 @@ async def _run_rows(
 
         async def invoke(row: DatasetRow) -> dict[str, Any]:
             await controller.acquire(config["provider"]["name"])
+            # One per row, not per run: rows generate concurrently against the
+            # same tool map, so a shared recorder would splice their calls.
+            recorder = TrajectoryRecorder()
+            row_tool_handlers = recorder.wrap(tool_handlers)
             started = datetime.now(UTC)
             started_clock = time.perf_counter()
             try:
                 result = await handler(
-                    config, row.input, tool_handlers, dict(row.variables)
+                    config, row.input, row_tool_handlers, dict(row.variables)
                 )
                 if not isinstance(result, Mapping):
                     raise TypeError("handler result must be a mapping")
@@ -644,6 +654,7 @@ async def invoke(row: DatasetRow) -> dict[str, Any]:
                     "generated_at": completed.isoformat().replace("+00:00", "Z"),
                     "latency_ms": round((time.perf_counter() - started_clock) * 1000),
                     "status": "COMPLETE",
+                    **row_fields(recorder),
                 }
                 usage = result.get("usage")
                 if isinstance(usage, Mapping):
@@ -663,6 +674,8 @@ async def invoke(row: DatasetRow) -> dict[str, Any]:
                     "latency_ms": round((time.perf_counter() - started_clock) * 1000),
                     "status": "ERROR",
                     "error": {"code": 5001, "message": f"handler raised: {error}"},
+                    # The calls that ran are what explain why it raised.
+                    **row_fields(recorder),
                 }
             finally:
                 controller.release()
@@ -776,25 +789,23 @@ def _judge_variables(
             ground_truth = parse_template(ground_truth, variables)
         elif expected is not None:
             ground_truth = str(expected)
-        # message_history carries FORMATTING_INSTRUCTIONS the same way the
-        # online path builds it (judges.run_judges), because that -- not the
-        # standalone formatting_instructions variable below -- is what every
-        # judge built from the AI Library's default templates (accuracy,
-        # relevance, toxicity, and any judge cloned from them) actually
-        # references. A judge authored before this variable existed must keep
-        # getting scored without edits.
+        # The calls the row made on its way to `output`, recorded during
+        # generation. Sits between input and output in message_history, which
+        # is where it happened.
+        trajectory = render_row_trajectory(row_result)
+        # Shared builder, not an inline join: this path and both online paths
+        # must show a judge the same conversation. The trajectory goes into
+        # message_history and nowhere else -- it is already the transcript
+        # variable judges read, and a second one would just let a rubric
+        # interpolate both and pay for the trajectory twice.
         variables.update(
             {
                 "input": row_result.get("input") or "",
                 "response_to_evaluate": output if output is not None else "",
-                "message_history": "\n\n".join(
-                    str(value)
-                    for value in (
-                        row_result.get("input"),
-                        output,
-                        FORMATTING_INSTRUCTIONS,
-                    )
-                    if value
+                "message_history": build_message_history(
+                    user_input=row_result.get("input"),
+                    trajectory=trajectory,
+                    output=output,
                 ),
                 "expected_output": expected if expected is not None else "",
                 "ground_truth_context": (

packages/client/src/launchdarkly_ai_server/graph.py

@@ -251,6 +251,7 @@ async def run_node(
                 base_track_data=result["track_data"],
                 tool_handlers=tool_handlers,
                 graph_key=key,
+                trajectory=result.get("trajectory", ""),
             )
 
             if from_node:
@@ -423,6 +424,7 @@ async def route(
                 base_track_data=result["track_data"],
                 tool_handlers=tool_handlers,
                 graph_key=key,
+                trajectory=result.get("trajectory", ""),
             )
 
             chosen_key = chosen()
@@ -490,6 +492,7 @@ async def stream_node(
         try:
             response = ""
             usage: dict[str, Any] = {"input": 0, "output": 0, "total": 0}
+            trajectory = ""
             track_data: TrackData = {
                 "runId": str(uuid.uuid4()),
                 "configKey": node.key,
@@ -528,6 +531,7 @@ async def stream_node(
                     response = event.get("response", "")
                     usage = event.get("usage") or usage
                     track_data = event.get("track_data") or track_data
+                    trajectory = event.get("trajectory") or trajectory
 
             judge_results = await run_judges(
                 config=node.config,
@@ -539,6 +543,7 @@ async def stream_node(
                 base_track_data=track_data,
                 tool_handlers=tool_handlers,
                 graph_key=key,
+                trajectory=trajectory,
             )
 
             if from_node:
@@ -624,6 +629,7 @@ async def stream_route(
         try:
             response = ""
             usage: dict[str, Any] = {"input": 0, "output": 0, "total": 0}
+            trajectory = ""
             track_data: TrackData = {
                 "runId": str(uuid.uuid4()),
                 "configKey": node.key,
@@ -664,6 +670,7 @@ async def stream_route(
                     response = event.get("response", "")
                     usage = event.get("usage") or usage
                     track_data = event.get("track_data") or track_data
+                    trajectory = event.get("trajectory") or trajectory
 
             # Judge against the node's original config, not the routing-augmented one.
             judge_results = await run_judges(
@@ -676,6 +683,7 @@ async def stream_route(
                 base_track_data=track_data,
                 tool_handlers=tool_handlers,
                 graph_key=key,
+                trajectory=trajectory,
             )
 
             chosen_key = chosen()
@@ -905,8 +913,8 @@ async def invoke(
             raise ValueError(f'Agent graph "{self._key}" is disabled')
 
         tracer = trace.get_tracer("@launchdarkly/ai-server")
-        with tracer.start_as_current_span("ld.ai.graph") as span:
-            span.set_attribute("ld.ai.graph.key", self._key)
+        with tracer.start_as_current_span("launchdarkly.graph") as span:
+            span.set_attribute("launchdarkly.graph.key", self._key)
 
             start_time = time.monotonic()
             path: list[str] = []
@@ -1115,8 +1123,8 @@ async def _stream_events(
             raise ValueError(f'Agent graph "{self._key}" is disabled')
 
         tracer = trace.get_tracer("@launchdarkly/ai-server")
-        span = tracer.start_span("ld.ai.graph", context=caller_context)
-        span.set_attribute("ld.ai.graph.key", self._key)
+        span = tracer.start_span("launchdarkly.graph", context=caller_context)
+        span.set_attribute("launchdarkly.graph.key", self._key)
         span_context = set_span_in_context(span, caller_context)
         ended: set[int] = set()
         start_time = time.monotonic()