docs: plan provider connection model lists - #5987
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Organization UI (inherited) Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (9)
Disabled knowledge base sources:
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdded eight design documents for provider connections. They define the data model, provider discovery, user interface, staged implementation plan, current-state research, and project status. ChangesProvider connections design
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Add experience.md and the clickable prototype, align the plan with the drawer, connection card, picker, and settings table, rename default-active models to recommended models, and record three new open decisions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
||
| The founder-provided design in [experience.md](experience.md) fixes the target interface. One term, | ||
| "Model providers", covers the playground banner, the picker footer, the drawer title, and the | ||
| Settings tab, which is renamed from "LLMs". The model picker lists connections rather than vendors. |
There was a problem hiding this comment.
Let's use AI providers instead of llms and model providers
There was a problem hiding this comment.
Done. The surface is named "AI providers" everywhere: banner, picker footer, drawer title, and the Settings tab (renamed from "LLMs"). Applied across all docs in e8b9d85.
| The founder-provided design in [experience.md](experience.md) fixes the target interface. One term, | ||
| "Model providers", covers the playground banner, the picker footer, the drawer title, and the | ||
| Settings tab, which is renamed from "LLMs". The model picker lists connections rather than vendors. | ||
| One drawer component serves both the playground and Settings. Subscriptions stay |
There was a problem hiding this comment.
there are minor differences from what is shown depending the context in the drawer. the drawer when opened from settings does not show the existing connections since they are seen in the table and can be configured there (opening directly the drawer wiht the config for the connection)
There was a problem hiding this comment.
Done. context.md and experience.md now state that the drawer opened from Settings shows the catalog only, with no Connected section, and that a Settings table row opens the connection card directly with that connection's configuration.
|
|
||
| ## Naming | ||
|
|
||
| Use one term everywhere: "Model providers". It appears in the playground banner call to action, the |
There was a problem hiding this comment.
Done, renamed throughout. The prototype still says "Model providers"; experience.md notes the review comment wins over the prototype.
| | DeepInfra | No, not through its public catalog | Catalog only through `GET /models/list` | Refresh models. Do not report that the key is valid. | | ||
| | Perplexity | No, not through its public model endpoint | Catalog only through `GET /v1/models` | Refresh models. Do not report that the key is valid. | | ||
| | MiniMax | Not confirmed in its public API reference | No confirmed list endpoint | Keep Agenta's catalog and manual IDs. Do not use paid generation as a test. | | ||
| | Aleph Alpha | Not confirmed in its current public API reference | No confirmed list endpoint | Keep Agenta's catalog and manual IDs. | |
There was a problem hiding this comment.
lets remove aleph alpha from the list since the startup is dead
There was a problem hiding this comment.
Done. Aleph Alpha is removed from the catalog and the discovery table. Existing stored records keep resolving so old configurations do not break, but the catalog no longer offers it.
| | MiniMax | Not confirmed in its public API reference | No confirmed list endpoint | Keep Agenta's catalog and manual IDs. Do not use paid generation as a test. | | ||
| | Aleph Alpha | Not confirmed in its current public API reference | No confirmed list endpoint | Keep Agenta's catalog and manual IDs. | | ||
| | Anyscale | No current hosted-model endpoint confirmed | No current hosted-model endpoint confirmed | Treat a user endpoint as custom. Keep manual IDs. | | ||
| | OpenAI-compatible endpoint | Not guaranteed by the compatibility label | Try `GET {base_url}/models` | A 404 or 405 means discovery is unsupported, not that the key is invalid. | |
There was a problem hiding this comment.
this mean we should allow manual addition of models in any case
There was a problem hiding this comment.
Done. New section "Manual model IDs are always available": every connection allows manual model identifiers for every provider in every state. Discovery is a convenience on top of manual entry, never a gate. This also resolves the providers with no free credential test.
| means one free read request can validate the credential and return models. `Catalog only` means the | ||
| request returns a provider-wide catalog and does not prove access to every returned model. | ||
|
|
||
| | Provider | Non-generation credential test | Model refresh | Proposed behavior | |
There was a problem hiding this comment.
you are missing aws, google gemini, and azure from the list
There was a problem hiding this comment.
don't forget their config is different
There was a problem hiding this comment.
Done. AWS Bedrock, Azure OpenAI, and Google Vertex AI are now first-class rows in the discovery table and the catalog (Google Gemini was already present as a standard API-key provider). Each has an honest credential-test and discovery row: Bedrock via ListFoundationModels, Azure via a deployments read on the user's endpoint, Vertex via a publisher-model list with OAuth from the service-account JSON.
There was a problem hiding this comment.
Covered. Their cards render provider-specific field sets instead of one API key field: Bedrock takes region plus bearer token or access-key pair, Azure takes key, endpoint, and API version, Vertex takes project, location, and service-account JSON. These field sets already exist in providerFields.ts, so the card is schema-driven by provider kind.
|
|
||
| ```text | ||
| provider models models the remote endpoint reports now | ||
| recommended models small curated list pre-checked when a connection is first created |
There was a problem hiding this comment.
default models (the ones that are selected per default)
There was a problem hiding this comment.
Done. The concept is renamed to "default models": the models that are selected per default. The card tags them "default" and pre-checks them on first fetch.
| catalog. Do not copy them into every connection. A missing connection selection means apply these | ||
| recommendations. A saved list, including an empty list, means use exactly what the user saved. | ||
|
|
||
| The interface tags these models "recommended", never "default". The connection card pre-checks them |
There was a problem hiding this comment.
why recommended not default
There was a problem hiding this comment.
if good reason i am ok with that
There was a problem hiding this comment.
The draft's reason was to signal curation rather than an unchosen fallback, but that distinction was not worth a second term. Renamed to "default models" everywhere per your comment.
|
|
||
| | Provider | Recommended model identifiers | | ||
| | --- | --- | | ||
| | OpenAI | `openai/gpt-5.5`, `openai/gpt-5.6-luna` | |
There was a problem hiding this comment.
no bad, gpt5.6 luna, terra and sol are default
There was a problem hiding this comment.
Done. OpenAI defaults are now openai/gpt-5.6-luna, openai/gpt-5.6-terra, and openai/gpt-5.6-sol. All three identifiers verified against the generated Pi catalog.
| | Provider | Recommended model identifiers | | ||
| | --- | --- | | ||
| | OpenAI | `openai/gpt-5.5`, `openai/gpt-5.6-luna` | | ||
| | Anthropic | `anthropic/claude-sonnet-5`, `anthropic/claude-haiku-4-5` | |
There was a problem hiding this comment.
Done. Anthropic defaults are now claude-fable-5, claude-opus-5, claude-sonnet-5, and claude-haiku-4-5. One caveat recorded in the doc: anthropic/claude-opus-5 does not exist in the pinned Pi catalog yet (it tops out at claude-opus-4-8), so the implementation must refresh the generated catalog via sync-model-catalog before that identifier resolves.
| | Groq | `groq/openai/gpt-oss-120b`, `groq/llama-3.1-8b-instant` | | ||
| | MiniMax | `minimax/MiniMax-M3`, `minimax/MiniMax-M2.7-highspeed` | | ||
| | Together AI | `together_ai/moonshotai/Kimi-K2.7-Code`, `together_ai/openai/gpt-oss-120b` | | ||
| | OpenRouter | `openrouter/~anthropic/claude-sonnet-latest`, `openrouter/~openai/gpt-mini-latest` | |
There was a problem hiding this comment.
very bad list, check most used openrouter model, include glm-5.2 and deepseek pro and flash latest
There was a problem hiding this comment.
Done. The OpenRouter defaults now follow the August 2026 usage rankings: z-ai/glm-5.2, deepseek/deepseek-v4-flash, deepseek/deepseek-v4-pro, openai/gpt-5.6-luna, xiaomi/mimo-v2.5, and tencent/hy3. All identifiers verified against the generated Pi catalog. The floating ~vendor aliases from the earlier draft are gone.
|
|
||
| Do not add a third secret kind in the first change. Do not convert existing records. | ||
|
|
||
| ## Standard provider record |
There was a problem hiding this comment.
waht about aws, azure, gemini, the data model should be general to allow them
There was a problem hiding this comment.
Done. data-model.md now has a section "The model is general". Simple API-key providers (including Gemini, already a standard kind) use provider_key records. Structured-credential providers (Azure, Bedrock, SageMaker, Vertex) use the existing custom_provider record whose kinds and per-kind field sets already exist. Both record types gain the same optional models and harnesses fields, and the connection card is schema-driven by provider kind, so a future provider means a new field set, not a new record type.
| - Search returns models across all connections with the connection as a subtitle. | ||
| - Existing agents continue to resolve through the compatibility path. | ||
|
|
||
| ### Later work: runtime registration and richer discovery |
There was a problem hiding this comment.
two things are missing from the plan, how this connects / to the harness in the case of agent and to the model in case of llm as a judge and prompt/completion/chat
All these routes should work and be updated and are part of the acceptance thing
There was a problem hiding this comment.
Done. The plan now has a fourth pull request wiring prompts, completion, chat, and LLM-as-a-judge to connections. Research confirmed both share one resolver (SecretsManager.get_provider_settings_from_workflow) and one model catalog, both separate from the agent path, and both break with two keys for one provider. PR 3 owns the agent-harness wiring, PR 4 makes the prompt-side catalog connection-aware, persists the connection slug beside the model, and teaches SecretsManager slug-first resolution with a family fallback. All routes are acceptance criteria in plan.md and context.md.
Rename the surface to AI providers, remove Aleph Alpha from the catalog, add AWS Bedrock, Azure OpenAI, and Vertex AI with their own field sets, make manual model IDs always available, rename the pre-checked list to default models with the corrected identifier lists, generalize the data model notes, and add pull request 4 wiring prompts, completion, chat, and LLM-as-a-judge to connections. status.md records which decisions the review resolved and which were taken for review.
|
Implementation of this plan is now open as a stacked series: #5995 (pull request 1, the vault connection contract), #5994 (the runner subscription status path from #5985), and #6001 (pull requests 2, 3, and 4: the settings experience, the connection-first picker, and the prompt/completion/chat/LLM-as-a-judge wiring, as annotated commits). The decisions taken autonomously during implementation are listed on #6001 with revert paths. |
dosubot
Bot
added
the
size:XXL
1 participant
Standard provider keys and custom providers expose different configuration shapes today. Standard
keys cannot save model or harness choices, while custom providers already own a model list. The
current frontend and resolver also assume one standard key per provider, which prevents two named
OpenAI connections from being selected independently.
This planning workspace proposes a compatibility-first path. It keeps both existing vault record
types, adds optional shared fields, and normalizes them into one provider-connection concept.
Existing records remain valid and require no data conversion.
Revision 2 (2026-08-12) addresses every review comment on this PR:
connection card directly.
credential field sets. The data model stays general: they use the existing custom-provider
record kinds, which already exist in the backend enums, frontend field catalog, and icon map.
lists: OpenAI is GPT-5.6 Luna/Terra/Sol; Anthropic adds Opus 5 and Fable 5 (with a noted
catalog-refresh dependency for Opus 5); OpenRouter follows the August 2026 usage rankings with
GLM-5.2 and DeepSeek V4 Pro and Flash.
Research confirmed those routes share one credential resolver and one stale model catalog, both
separate from the agent path; the plan makes them connection-aware with a slug-first resolver
and a family fallback. All run routes are acceptance criteria.
status.mdseparates decisions resolved by the review, decisions taken during this revision forfounder review (Done gating for untestable providers, pin-versus-follow for the default set, the
extra OpenRouter picks, no pre-checked defaults for Bedrock/Azure/Vertex), and the remaining open
decisions.
The settings design needs two distinct backend results behind its single Test action. A credential
test says whether the provider accepted the credential. Model discovery says which model
identifiers the provider returned. Public catalogs such as Perplexity's can refresh models but
cannot prove that a key works. The plan prevents a false
Key validstate and keeps Agenta'scurrent catalog when discovery is unsupported.
The work is split into four dependent changes:
model refresh, and manual model IDs.
persist the exact connection slug.
How to review
context.mdfor the current experience, goal, and non-goals.experience.mdfor the founder-provided interface design, and click throughmodel-providers-ux-prototype.htmlin a browser. The prototype is a sketch, not referencecode, and its "Model providers" naming is superseded.
research.mdfor the existing storage, resolver, and catalog behavior, including theprompt and LLM-as-a-judge paths.
provider-discovery.mdfor provider support, honest status messages, fallback behavior,and the default-model lists.
data-model.mdfor identity, defaults, and the shared connection shape.plan.mdfor the four pull requests and acceptance checks.status.mdfor resolved decisions, decisions taken for review, and open decisions.Checks
git diff --check(
sdks/python/agenta/sdk/agents/data/pi_models.generated.json), with the Opus 5 gap noted.