Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/splunk_ao/converter/attribute_mapping.py
Original file line number Diff line number Diff line change
Expand Up @@ -447,6 +447,10 @@ def _set_orchestration_content(attrs: MutableMapping[str, AttributeValue], span:
return
if full_history and input_messages is not None and output_messages[: len(input_messages)] == input_messages:
output_messages = output_messages[len(input_messages) :]
# Full history includes all messages in the run, not just the final response.
# Keep only the last message — it is always the agent's final output.
Comment on lines +450 to +451

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 minor (bug): "it is always the agent's final output" is not true, and the comment will mislead whoever touches this next.

The last message is only the final answer when the run terminated normally on an assistant text message. It is not when the run ends on a tool-call AIMessage — e.g. interrupt_before=["tools"], a hit recursion limit, or an aborted run. In that case content is "" and _mapped_message (line 190) produces parts == [tool_call_part] with no text part, so the Output column still renders and the preceding assistant text that was present is now discarded as well. The bug this PR sets out to fix therefore survives for those runs.

At minimum, soften the comment to describe the heuristic rather than assert an invariant. If you want the interrupted-run case actually covered, prefer the last message that yields a non-empty text part, falling back to the last message — and add a test for it.

Suggested change
# Full history includes all messages in the run, not just the final response.
# Keep only the last message — it is always the agent's final output.
# Full history includes every message in the run, not just the final response. Child spans
# already carry the intermediate steps, so surface only the last message here. Note this is a
# heuristic: a run that ends on a tool-call message has no final text to show.

🤖 Generated by the Astra agent

if full_history and len(output_messages) > 1:
output_messages = [output_messages[-1]]
Comment on lines 448 to +453

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major (bug): Silent message loss on non-root LangGraph node spans.

full_history is True for any output that parses to a mapping with a top-level "messages" list of role-bearing dicts (_message_container_is_message_sequence, lines 220-228). It does not mean "this output is accumulated history" — only "this output is a message container". Since on_chain_end records every chain's raw return value (handlers/langchain/handler.py:119-125) and base_handler.py:145-157 emits non-root chains as WorkflowSpans, node-level state updates land here too.

Concrete failure: LangGraph's prebuilt ToolNode returns one ToolMessage per tool call, so a turn with two parallel tool calls produces node output {"messages": [tool_msg_1, tool_msg_2]}. Serialized messages always carry role (utils/serialization.py:245), so full_history is True. The dedup on line 448 cannot fire (2 messages vs. the N-message input state), so line 452 fires and gen_ai.output.messages for that invoke_workflow tools span becomes [tool_msg_2]tool_msg_1 is dropped from telemetry entirely. This repo ships examples that hit exactly this shape: examples/agent/langgraph-traceloop/agent.py:180 (ToolNode) and the create_react_agent FSI examples, both used with SplunkAOCallback.

The ticket's own wording is "After dedup removes the input prefix, multiple messages remain" — so gate on the dedup actually having matched. That preserves the fix for the reported case and leaves genuine multi-message node updates intact.

One caveat to confirm: if a graph ever prepends or trims messages so the input is no longer a prefix, the dedup won't match and the root span won't be reduced. If that is a real shape for your app, prefer reducing at the handler layer for the root agent node instead (mirroring middleware.py:199-202).

Suggested change
if full_history and input_messages is not None and output_messages[: len(input_messages)] == input_messages:
output_messages = output_messages[len(input_messages) :]
# Full history includes all messages in the run, not just the final response.
# Keep only the last message — it is always the agent's final output.
if full_history and len(output_messages) > 1:
output_messages = [output_messages[-1]]
history_stripped = False
if full_history and input_messages is not None and output_messages[: len(input_messages)] == input_messages:
output_messages = output_messages[len(input_messages) :]
history_stripped = True
# The stripped remainder is the run's intermediate steps plus the final response. Child
# LLM/tool spans already carry the intermediates, so surface only the final message here.
if history_stripped and len(output_messages) > 1:
output_messages = [output_messages[-1]]

🤖 Generated by the Astra agent

attrs["gen_ai.output.messages"] = _json_string(_with_finish_reasons(output_messages))


Expand Down
84 changes: 84 additions & 0 deletions tests/test_attribute_mapping.py
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,90 @@ def test_orchestration_output_omits_repeated_input_history() -> None:
]


def test_orchestration_full_history_with_tool_call_keeps_last_message() -> None:
# LangGraph accumulated state: user → tool-call AI (empty content) → tool response → final AI
# The first post-dedup message has empty content; the UI would show "—" without the fix.
user = {"role": "user", "content": "What is the dosage of Lisinopril?"}
ai_toolcall = {"role": "assistant", "content": "", "tool_calls": [{"id": "tc1", "function": {"name": "rag_search", "arguments": '{"query":"Lisinopril dosage"}'}}]}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 nit (other): ruff format --check reports this file would be reformatted — lines 425, 469, 471, and 473 exceed the 120-char limit. .pre-commit-config.yaml runs the ruff-format hook, so this will fail pre-commit. Running ruff format tests/test_attribute_mapping.py splits these four dict literals across lines.

🤖 Generated by the Astra agent

tool_resp = {"role": "tool", "content": "Lisinopril: 10mg daily", "tool_call_id": "tc1"}
ai_final = {"role": "assistant", "content": "Common dosage is 10mg once daily."}

span = AgentSpan(
name="Agent",
agent_type=AgentType.default,
input=json.dumps({"messages": [user]}),
output=json.dumps({"messages": [user, ai_toolcall, tool_resp, ai_final]}),
)

attrs = build_span_attributes(span)

output_messages = json.loads(attrs["gen_ai.output.messages"])
assert len(output_messages) == 1
assert output_messages[0]["role"] == "assistant"
assert output_messages[0]["parts"][0]["content"] == "Common dosage is 10mg once daily."


def test_orchestration_full_history_multi_turn_keeps_last_message() -> None:
# Multi-turn: output contains the full conversation history after multiple exchanges.
# Only the last message should be kept regardless of role.
user1 = {"role": "user", "content": "Hello"}
ai1 = {"role": "assistant", "content": "Hi, how can I help?"}
user2 = {"role": "user", "content": "What is Lisinopril?"}
ai2 = {"role": "assistant", "content": "Lisinopril is a blood pressure medication."}

span = AgentSpan(
name="Agent",
agent_type=AgentType.default,
input=json.dumps({"messages": [user1]}),
output=json.dumps({"messages": [user1, ai1, user2, ai2]}),
)

attrs = build_span_attributes(span)

output_messages = json.loads(attrs["gen_ai.output.messages"])
assert len(output_messages) == 1
assert output_messages[0]["parts"][0]["content"] == "Lisinopril is a blood pressure medication."


def test_orchestration_full_history_multiple_tool_rounds_keeps_last_message() -> None:
# Two tool call rounds before the final answer — last message is still the only output.
user = {"role": "user", "content": "Compare Lisinopril and Amlodipine"}
tc1_ai = {"role": "assistant", "content": "", "tool_calls": [{"id": "tc1", "function": {"name": "search", "arguments": '{"query":"Lisinopril"}'}}]}
tc1_resp = {"role": "tool", "content": "Lisinopril: ACE inhibitor", "tool_call_id": "tc1"}
tc2_ai = {"role": "assistant", "content": "", "tool_calls": [{"id": "tc2", "function": {"name": "search", "arguments": '{"query":"Amlodipine"}'}}]}
tc2_resp = {"role": "tool", "content": "Amlodipine: calcium channel blocker", "tool_call_id": "tc2"}
ai_final = {"role": "assistant", "content": "Lisinopril is an ACE inhibitor; Amlodipine is a calcium channel blocker."}

span = AgentSpan(
name="Agent",
agent_type=AgentType.default,
input=json.dumps({"messages": [user]}),
output=json.dumps({"messages": [user, tc1_ai, tc1_resp, tc2_ai, tc2_resp, ai_final]}),
)

attrs = build_span_attributes(span)

output_messages = json.loads(attrs["gen_ai.output.messages"])
assert len(output_messages) == 1
assert "Amlodipine" in output_messages[0]["parts"][0]["content"]
Comment on lines +421 to +486

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major (testing): Missing coverage for the cases where this change is most likely to be wrong. All three new positive tests are AgentSpan with a clean user → tool-call → tool → final-AI history, which is the one shape known to work. Please add:

  1. WorkflowSpan with a multi-message node update that is not accumulated history — e.g. output=json.dumps({"messages": [tool_msg_1, tool_msg_2]}) with an unrelated input state. This is the parallel-tool-call ToolNode shape and currently loses tool_msg_1; it is the regression flagged on attribute_mapping.py:448.
  2. Last message is a tool-call assistant with content="" (interrupted / recursion-limited run) — asserts what the exported output looks like when the heuristic does not hold.
  3. Last message has role: "tool" — the multi-turn test comments "regardless of role" but never actually exercises a non-assistant final message.

Also note test_orchestration_full_history_multiple_tool_rounds_keeps_last_message (line 467) is nearly a duplicate of the first test — it adds a second tool round but no new branch. Consider folding the two into a pytest.mark.parametrize and spending the saved test on case 1 above, which is the one that actually catches a defect.

Separately: test_orchestration_output_omits_repeated_input_history (line 403) is now non-discriminating for the prefix-dedup on lines 448-449 — the reduction alone produces the same single assistant message, so deleting the dedup would not fail it. Gating the reduction on the dedup (see the line 448 comment) restores that test's value.

🤖 Generated by the Astra agent



def test_orchestration_non_full_history_output_not_reduced() -> None:
# Plain string output (full_history=False) — the last-message reduction must NOT fire.
span = AgentSpan(
name="Agent",
agent_type=AgentType.default,
input="What is Lisinopril?",
output="Lisinopril is a blood pressure medication.",
)

attrs = build_span_attributes(span)

output_messages = json.loads(attrs["gen_ai.output.messages"])
assert len(output_messages) == 1
assert output_messages[0]["parts"][0]["content"] == "Lisinopril is a blood pressure medication."
Comment on lines +489 to +502

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major (testing): This test is vacuous — it passes identically with or without the change, so it does not guard the full_history gate it claims to.

Tracing a plain string through _message_container: _parse_json_string fails and returns the string, _mapping_view returns None, and the Sequence branch at attribute_mapping.py:234 explicitly excludes str. So it returns (value, False, False) and _message_sequence yields exactly one message. Both full_history and len(output_messages) > 1 are independently false, so the new branch could never fire for this input. The len == 1 assertion is satisfied by the input shape, not by the guard.

To actually exercise the guard, use an input where the reduction would fire if the gate were dropped — a bare JSON list of messages, which _message_container:234-236 classifies as full_history=False while still producing multiple messages:

Suggested change
def test_orchestration_non_full_history_output_not_reduced() -> None:
# Plain string output (full_history=False) — the last-message reduction must NOT fire.
span = AgentSpan(
name="Agent",
agent_type=AgentType.default,
input="What is Lisinopril?",
output="Lisinopril is a blood pressure medication.",
)
attrs = build_span_attributes(span)
output_messages = json.loads(attrs["gen_ai.output.messages"])
assert len(output_messages) == 1
assert output_messages[0]["parts"][0]["content"] == "Lisinopril is a blood pressure medication."
def test_orchestration_bare_message_list_output_not_reduced() -> None:
# Given: output is a bare message list (full_history=False), not a {"messages": [...]} container.
# When: attributes are built.
# Then: every message survives — the last-message reduction must not fire.
messages = [
{"role": "assistant", "content": "First"},
{"role": "assistant", "content": "Second"},
]
span = AgentSpan(
name="Agent",
agent_type=AgentType.default,
input="What is Lisinopril?",
output=json.dumps(messages),
)
attrs = build_span_attributes(span)
output_messages = json.loads(attrs["gen_ai.output.messages"])
assert len(output_messages) == 2
assert output_messages[0]["parts"][0]["content"] == "First"
assert output_messages[1]["parts"][0]["content"] == "Second"

🤖 Generated by the Astra agent



def test_orchestration_preserves_schema_valid_parts_and_tool_calls() -> None:
span = WorkflowSpan(
name="tool-workflow",
Expand Down
Loading