Skip to content

feat(AIC-3449): Support pulling in configs for evals from code - #108

Merged
aknight-ld merged 4 commits into
mainfrom
AIC-3449-support-config-calls
Sep 24, 2026
Merged

aknight-ld merged 4 commits into
mainfrom
AIC-3449-support-config-calls

Conversation

@aknight-ld

@aknight-ld aknight-ld commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Current build of evaluations from the SDK requires the user to define a generation dict by hand. Callers who want to evaluate a variation that already exists have to copy its model, prompt, and tools by hand. This PR provides functionality to let them name ai_config + variation, fetch that variation's definition, and treat any explicit argument as an override.


Note

Overview
Evaluations run() can seed settings from an existing AI Config variation instead of requiring a fully hand-built generation dict. Callers pass ai_config and variation; the SDK fetches the latest variation (and linked model config) via the management API, builds defaults for model/provider/parameters, prompt, tools, and judges, then applies optional generation, tools, and criteria as overrides.

generation is now optional when both config keys are provided. _merge_generation layers overrides field-by-field (parameters merge by key; supplying instructions or messages replaces both fetched prompt slots). Validation is split into config-source checks (before any I/O) and post-merge generation checks. Variation-attached tools require implementations or the run fails early; attached judges become default criteria when criteria is omitted. Warnings are logged when a variation pins an older tool version than the API’s latest.

Adds AIConfigVariation parsing and runner helpers _fetch_config_variation / _fetch_model_config, plus broad tests for seeding, overrides, validation, and edge cases.

Reviewed by Cursor Bugbot for commit 510d21d. Bugbot is set up for automated code reviews on this repo. Configure here.

@donei003 donei003 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. But had some feedback which we can incorporate

Comment thread packages/client/src/launchdarkly_ai_server/evaluations/module.py Outdated
Comment thread packages/client/src/launchdarkly_ai_server/evaluations/types.py Outdated
)
provider: Any = None
model_config_key = latest.get("modelConfigKey")
if isinstance(model_config_key, str) and model_config_key:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would happen if this wasn't true?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this check represents validation to make sure that if a model config is set, it's set correctly. I'm restructuring it to be clearer and adding an exception to fail loudly

Comment thread packages/client/src/launchdarkly_ai_server/evaluations/runner.py Outdated
@aknight-ld
aknight-ld force-pushed the AIC-3449-support-config-calls branch from c49366c to 39a7e2e Compare September 24, 2026 19:59

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 39a7e2e. Configure here.

Comment thread packages/client/src/launchdarkly_ai_server/evaluations/runner.py
@aknight-ld
aknight-ld merged commit fa374e2 into main Sep 24, 2026
7 checks passed
@aknight-ld
aknight-ld deleted the AIC-3449-support-config-calls branch September 24, 2026 20:14
@github-actions github-actions Bot mentioned this pull request Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants