feat(evaluations): accept inline tool definitions in run(tools=...)

[donei003]

Lets a developer define a tool in code instead of creating it in LaunchDarkly first.

_resolve_tools (evaluations/runner.py) currently requires every key in the tools map to resolve via GET projects/<p>/ai-tools/<key>, and that response is the only source for the handler config's tool description and parameters. There is no slot in the public API for a caller-supplied body.

Changes

• New exported InlineTool type carrying description, schema, and the executable (implementation), accepted in the same tools map as today's bare callables. A bare callable still means "library tool, resolve by key" — no change for existing callers, and a map may mix both sources.
• _resolve_tools skips the ai-tools GET for inline entries.
• Validation before any network I/O (_validate_tools, called from run() ahead of every request): key non-blank, schema a JSON object and JSON-serializable with allow_nan=False, implementation callable, no duplicate keys across sources, and an inline key that also names a library tool is rejected naming the key.
• Eval create body sends {key, schema, description, source: "inline"} for inline entries and {key, version, source: "library"} for library entries. tools is still omitted entirely when the map is empty.
• Handler config synthesis is unchanged in shape — config["tools"][key] = {description, parameters} is fed from the inline body instead of the GET response, and an InlineTool is unwrapped to its executable before any handler sees it, so handlers see no difference.

Decision for reviewers

Whether a NativeTool value may be paired with an inline definition. A native tool has no schema of its own, so I've made the combination a validation error rather than inventing a meaning for it.

One other thing worth a look: the brief asked for "no duplicate keys across sources" and "an inline key that also names a library tool is rejected". Because both sources share one tools mapping, a plain dict cannot express the same key twice, so those checks needed a concrete surface. Implemented as: (a) the map is validated as the caller's Mapping before it is copied into a dict, so a Mapping that yields a key twice is caught rather than silently collapsed; and (b) keys are compared case-insensitively only when an inline definition is involved — {"lookup_order": fn, "Lookup_Order": InlineTool(...)} is rejected naming the key, while two library keys differing only by case remain two library lookups, since changing that would alter existing library behavior. Both are tested. Say the word if you'd rather the collision rule were exact-case only, or extended to library/library pairs.

Verification

Run in a clean worktree off origin/main:

• uv run pytest — 1318 passed, 11 skipped (full monorepo suite).
• uv run ruff check . — all checks passed.
• uv run ruff format --check . — 114 files already formatted.
• uv run mypy packages/*/src (the repo's make typecheck, which CI mirrors) — no issues in 50 source files. Note the repo does not type-check test suites under strict, per AGENTS.md, so the new tests are not mypy-covered.

New tests in packages/client/tests/test_evaluations_run.py:

• test_inline_tool_runs_without_reading_the_tool_api — asserts the inline request sequence in full as (method, path) pairs, and asserts explicitly that no recorded URL contains /ai-tools, rather than relying on the strict-sequence transport to trip on a surplus request. Also asserts the create body's inline entry, the synthesized config["tools"], and that the handler receives the bare callable.
• test_inline_tool_description_defaults_to_empty_string
• test_mixed_library_and_inline_tools_each_keep_their_own_source — exactly one tool GET, for the library key only; both wire entries; both config entries.
• test_bad_tool_entry_is_rejected_with_zero_requests — parametrized over blank inline key, blank library key, schema=None, schema as a list, non-serializable schema, NaN, Infinity, non-callable implementation, non-string description, and an entry that is none of the three. Each asserts transport.requests == [].
• test_native_tool_paired_with_an_inline_definition_is_rejected — zero requests.
• test_native_tool_on_its_own_still_resolves_from_the_library — the library path for NativeTool is unchanged.
• test_inline_key_that_also_names_a_library_tool_is_rejected — zero requests.
• test_two_library_keys_differing_only_by_case_are_still_two_lookups — pins the library path against the new collision rule.
• test_duplicate_tool_key_across_sources_is_rejected_with_zero_requests — uses a custom Mapping that yields one key twice.

One existing assertion changed: the happy-path create body in test_complete_run_with_zero_failed_and_error_rows_passes now expects {"key": "lookup_order", "version": 7, "source": "library"}.

CI on this branch is green: Lint & format, Type check, Tests, Build all packages, Install & sync all pass.

Not verified: nothing was exercised against a real gonfalon proxy or ai-evaluator — the source discriminator and the inline body shape are asserted against the recording fake transport only.

Dependencies

Requires the gonfalon proxy/API change deployed. Cross-language spec is ai-sdks-monorepo TESTING.md §8 (spec PR open concurrently).

🤖 Generated with Claude Code