Add model discovery integration tests without managed config - #662
Merged
Merged
Conversation
andy-xu-db
force-pushed
the
andy/model-discovery-integration-unmanaged
branch
from
September 16, 2026 15:02
5204d14 to
9751ad8
Compare
andy-xu-db
changed the base branch from
main
to
codex/model-discovery-test-stack
September 16, 2026 15:02
andy-xu-db
force-pushed
the
codex/model-discovery-test-stack
branch
from
September 16, 2026 18:32
d85847c to
2c0b3df
Compare
andy-xu-db
force-pushed
the
andy/model-discovery-integration-unmanaged
branch
from
September 16, 2026 18:32
9751ad8 to
88b4d6f
Compare
andy-xu-db
changed the base branch from
codex/model-discovery-test-stack
to
andy/model-discovery-integration-tests
September 16, 2026 20:06
andy-xu-db
force-pushed
the
andy/model-discovery-integration-unmanaged
branch
3 times, most recently
from
September 17, 2026 20:03
e907761 to
e4a7d5c
Compare
andy-xu-db
force-pushed
the
andy/model-discovery-integration-tests
branch
from
September 17, 2026 21:37
8cc5eb8 to
ecd7c40
Compare
andy-xu-db
force-pushed
the
andy/model-discovery-integration-unmanaged
branch
from
September 17, 2026 21:37
e4a7d5c to
5404b69
Compare
lilly-luo
approved these changes
Sep 17, 2026
andy-xu-db
force-pushed
the
andy/model-discovery-integration-unmanaged
branch
2 times, most recently
from
September 17, 2026 22:54
debbbfb to
6fb6e7f
Compare
andy-xu-db
force-pushed
the
andy/model-discovery-integration-unmanaged
branch
from
September 21, 2026 18:30
6fb6e7f to
394cbde
Compare
andy-xu-db
requested review from
AarushiShah-db and
rohita5l
as code owners
September 21, 2026 18:30
andy-xu-db
changed the base branch from
andy/model-discovery-integration-tests
to
main
September 21, 2026 18:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Align discovery integration coverage with the fetched
mainat6e5b97b(#775), merged in043ab5e. Do not restore the obsolete discovery feature stack or change runtime behavior.UG_ENABLE_MODEL_DISCOVERYswitch. Remove obsolete unmanaged disable scenarios and duplicate managed variants. Managed discovery and source-override rejection remain covered by repository Cases 1–6.ug configurehas no--model-locationoption. Cases 7–10 cover configured/fresh default launches: Claude automatically cachessystem.aimodels (including recognized Anthropic gateway aliases) and opens its picker; Codex leaves model/reasoning preferences unset and exposes its native catalog without a generated scoped file.--providerand--model-locationCases 11–14 with exact scoped catalog assertions. Codex's parent catalog must exactly match an independent, parent-scoped API response and include the dedicated Codex service; it must not assume the Claude service supports the Codex API. Claude's provider models can appear as exact native Haiku 4.5, Opus 5, or Sonnet 5 picker rows.Current coverage
Repository scenario numbers now run consecutively from 01 to 14, with configured/fresh variants sharing a number. Test names, artifact labels, both repository READMEs, and contract checks use the same mapping. The external Google doc and its numbering are untouched. The complete integration suite collects 97 executions: 58 live (25 Claude, 33 Codex), 27 managed-fixture, 4 managed-workspace, 1 workspace-switch, and 7 installation.
Discovery checks configure, list metadata, and open/close pickers without inference prompts. Separate existing task/agent tests do perform inference.
Latest live result and assertion fixes
At
7751cf1, all integration jobs passed in CI run 35655313004:Independent review found and fixed picker and catalog evidence gaps. Claude assertions now match numbered picker rows, not startup banners or footer text, and explicitly recognize native family/version deduplication. Managed caches/catalogs are compared with independent read-only provider-list responses. Codex parent expectations follow the API-compatible scoped response, rejecting missing, extra, duplicate, or out-of-schema app-server models.
Claude still accepts only recognized gateway aliases while preserving system.ai membership, raw-ID uniqueness, and discovered-family inclusion. Codex requires absent model/reasoning preferences for unmanaged native defaults. No production code, workspace configuration, or GitHub secrets were changed. The external Google doc remains untouched.
Verification
uv run --frozen pytest --ignore-glob='tests/test_e2e*.py'. Existing live-agent/e2e probes were explicitly excluded locally.--help,git diff --check, and collection of all 97 integration tests passed.