fix(eval): reject out-of-range scores from LLM judge tool calls

[mjnovice]

fix(eval): reject out-of-range scores from LLM judge tool calls

Problem

extract_tool_call_response in legacy_llm_helpers.py takes the score argument from the judge model's submit_evaluation tool call and passes it through with no range validation (score = float(arguments["score"])), even though the tool schema declares "Numeric score between 0 and 100".

gemini-2.5-flash occasionally emits corrupted numeric tool arguments (digit repetition). When that happens, the raw value flows through LegacyLlmAsAJudgeEvaluator → eval runtime → backend storage, and no layer on the serverless eval path validates it. A single corrupted item then poisons the run-level evaluator average.

Production proof

Eval run for the "Donor Eligibility Triage Agent" (64 items, default Gemini 2.5 + Sonnet trajectory evaluators). The run list UI showed the Gemini evaluator column as 15559.13% (screenshot in comments / Slack thread).

Trace 1 — trace-1784158433174.json (STANDARD - Thalassemia item), raw gemini-2.5-flash response:

{
  "name": "submit_evaluation",
  "arguments": {
    "justification": "The core meaning and all key facts match perfectly between the expected and actual outputs. ...",
    "score": 989898
  }
}

Trace 2 — trace-1784158524459.json (STANDARD - Topical Medications item), raw gemini-2.5-flash response:

{
  "name": "submit_evaluation",
  "arguments": {
    "score": 950,
    "justification": "The actual output is semantically equivalent to the expected output. All key facts are present and accurate. ..."
  }
}

Both justifications describe a near-perfect match — the model clearly intended ~98-100 and ~95. One item at 989,898 averaged over 64 items ≈ 15,559 — exactly the corrupted aggregate shown in the UI.

Note the trace's Evaluation output span shows the display value clamped to 100 (_spans.py normalize_score_to_100), which masks the corruption at item level while the raw value still poisons the run aggregate — that inconsistency is what made this hard to spot.

Repro

• Deterministic: pytest packages/uipath/tests/evaluators/test_legacy_llm_helpers.py — the new tests feed the exact production payloads (989898, 950) into extract_tool_call_response.
• End-to-end: run any eval set using a legacy LLM-judge evaluator (e.g. "Default Evaluator - Gemini 2.5") on gemini-2.5-flash over a reasonably large set; the corruption is stochastic (~2 out of ~128 judge calls in the run above). Point the evaluator at a mock LLM gateway returning score: 989898 to force it on the first item.

Fix

Reject invalid scores (out-of-range or non-finite) with a ValueError. The track_evaluation_metrics decorator on the legacy evaluators converts the exception into an ErrorEvaluationResult (score=0, ScoreType.ERROR) with the invalid value in the details, so the eval item surfaces as an evaluator error in the UI instead of silently recording a fabricated score.

Design note: an earlier revision clamped the score to 0-100 with a warning in the justification. Review feedback (Anirudh) rightly pointed out that clamping fabricates a score the model never gave (950 → 100 when the model likely meant 95), so this now fails the evaluation explicitly.

A companion defense-in-depth PR (UiPath/Agents#5831) makes the backend aggregation treat out-of-range stored values like error scores.

Testing

• packages/uipath/tests/evaluators/test_legacy_llm_helpers.py: 10 tests (production payloads 989898/950, negative, boundaries, NaN/±inf, missing field) — all pass.
• Full tests/evaluators/ suite: 507 passed, no regressions.
• ruff check, ruff format --check, mypy clean on changed files.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR hardens legacy LLM-judge evaluation parsing by validating and clamping tool-call score values to prevent corrupted model outputs from poisoning run-level aggregates, while making any corrections auditable in the UI.

Changes:

• Clamp finite out-of-range score values from submit_evaluation tool calls into [0, 100] and prepend a warning line to the justification.
• Reject non-finite scores (NaN/inf) with a ValueError.
• Add a focused pytest suite covering normal, boundary, out-of-range, negative, non-finite, and missing-field cases (including production payloads).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
packages/uipath/src/uipath/eval/evaluators/legacy_llm_helpers.py	Adds non-finite validation and clamps out-of-range judge scores, surfacing corrections in justification.
packages/uipath/tests/evaluators/test_legacy_llm_helpers.py	Adds regression tests for score parsing/clamping behavior using production-like payloads.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mjnovice]

Proof: raw gemini-2.5-flash submit_evaluation tool calls extracted from the production traces

Extracted from trace-1784158433174.json and trace-1784158524459.json (Model run spans of the "Default Evaluator - Gemini 2.5"):

[
  {
    "eval_item": "STANDARD - Thalassemia",
    "model": "gemini-2.5-flash",
    "tool_call": {
      "name": "submit_evaluation",
      "arguments": {
        "justification": "The core meaning and all key facts match perfectly between the expected and actual outputs. The actual output accurately describes the conditions for deferral related to Thalassemia, including minor or trait, major, and combined sickle cell and thalassemia traits. There is no information missing from the actual output, nor is there any incorrect or contradictory information. The differences are limited to formatting, such as the use of bullet points and bolding, and slight variations in phrasing, which do not alter the semantic meaning or factual accuracy. The actual output also includes the word \"Thalassemia\" at the beginning, which provides helpful context without introducing any inaccuracies.",
        "score": 989898
      }
    }
  },
  {
    "eval_item": "STANDARD - Topical Medications",
    "model": "gemini-2.5-flash",
    "tool_call": {
      "name": "submit_evaluation",
      "arguments": {
        "score": 950,
        "justification": "The actual output is semantically equivalent to the expected output. All key facts are present and accurate. The phrasing is slightly different but conveys the same meaning. Formatting differences and citations are ignored as per instructions."
      }
    }
  }
]

Both justifications describe a near-perfect match, so the intended scores were ~98-100 and ~95 — the numeric argument was corrupted by the model. UI screenshot of the resulting 15559.13% run aggregate attached below.

[github-actions]

🚨 Heads up: uipath-integrations cross-tests are FAILING 🚨

Your changes may break one or more integrations in uipath-integrations-python:

• uipath-openai-agents
• uipath-google-adk
• uipath-agent-framework
• uipath-llamaindex
• uipath-pydantic-ai

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud