Skip to content

fix(catalog): preserve provider capabilities on custom models - #963

Closed
MarcTCruz wants to merge 1 commit into
lidge-jun:devfrom
MarcTCruz:fix/custom-model-capability-metadata
Closed

fix(catalog): preserve provider capabilities on custom models#963
MarcTCruz wants to merge 1 commit into
lidge-jun:devfrom
MarcTCruz:fix/custom-model-capability-metadata

Conversation

@MarcTCruz

@MarcTCruz MarcTCruz commented Aug 3, 2026

Copy link
Copy Markdown

Summary

  • preserve registry-enriched provider capability hints when a custom model shadows a provider model row
  • keep explicit custom display name, context window, and modalities authoritative while retaining noVisionModels augmentation
  • prevent the catalog from synthesizing generic reasoning levels for custom models declared through noReasoningModels

Fixes #962

Verification

  • bun scripts/test.ts tests/catalog-custom-model-metadata.test.ts tests/catalog-vision-sidecar-modalities.test.ts — 13 passed, 0 failed
  • bun run typecheck — passed
  • bun run lint:gui — passed
  • bun run privacy:scan — passed
  • bun run doctor:gui:if-changed — skipped as expected because no GUI files changed
  • live CLI check with an isolated OPENCODEX_HOME — emitted reasoningEfforts: [] for the duplicated Ollama custom model
  • bun run test — 7,889 passed and 10 skipped; one pre-existing failure remained in native-profile-drain-server.test.ts (Live/Realtime sideband retains main ownership...). The same failure reproduces on clean dev at 1d1c8038.

Checklist

  • Scope is focused and avoids unrelated refactors.
  • Docs or release notes were considered; none are needed for this internal catalog-correctness regression.
  • Security-sensitive areas were considered; this change does not modify authentication, secret handling, or security defaults.

Summary by CodeRabbit

  • New Features
    • Custom catalog models now inherit provider capabilities and registry metadata when not explicitly configured.
    • Explicit custom context limits and input modalities remain unchanged.
    • Vision sidecars continue to add image support where applicable.
  • Bug Fixes
    • Improved handling of context limits, reasoning capabilities, and modality metadata for custom models.

@github-actions github-actions Bot added the bug Something isn't working label Aug 3, 2026
@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Custom catalog models now inherit provider capability metadata, including reasoning configuration and conditional context limits. Explicit custom context windows and input modalities remain authoritative. Tests cover metadata inheritance, vision-sidecar behavior, and catalog reasoning levels.

Changes

Catalog metadata inheritance

Layer / File(s) Summary
Provider metadata inheritance and custom overrides
src/codex/catalog/provider-fetch.ts, tests/catalog-custom-model-metadata.test.ts
Custom models apply provider capability hints while preserving explicit context and modality values. Tests verify inherited reasoning capabilities and empty catalog reasoning levels without a default.
Vision-sidecar metadata validation
tests/catalog-vision-sidecar-modalities.test.ts
Tests call gatherRoutedModels directly and verify image augmentation, preserved custom fields, and registry-provided reasoning efforts without a default effort.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant gatherRoutedModels
  participant ProviderRegistry
  participant CatalogEntries
  gatherRoutedModels->>ProviderRegistry: read provider capability hints
  ProviderRegistry-->>gatherRoutedModels: return reasoning and context metadata
  gatherRoutedModels->>CatalogEntries: provide merged custom model metadata
  CatalogEntries-->>gatherRoutedModels: expose reasoning levels and defaults
Loading

Possibly related PRs

Suggested reviewers: lidge-jun, ingwannu

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes preserving provider capabilities on custom catalog models, which is the primary change.
Linked Issues check ✅ Passed The implementation and tests preserve provider metadata while keeping custom fields authoritative and preventing synthesized reasoning levels for affected models [#962].
Out of Scope Changes check ✅ Passed The changes are limited to catalog provider merging and focused tests for custom metadata, vision sidecars, and reasoning capabilities.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/catalog-custom-model-metadata.test.ts`:
- Around line 16-17: Add noReasoningModels containing "qwen-coder-3b" to the
provider fixture alongside modelReasoningEfforts in the relevant test setup.
Preserve the existing assertions that this model exposes no supported reasoning
levels and no default reasoning level, ensuring the regression path verifies
inherited noReasoningModels metadata.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 66a9e7cf-6a0b-4ce5-b2d0-c031201639dc

📥 Commits

Reviewing files that changed from the base of the PR and between cb8fcf3 and 4666235.

📒 Files selected for processing (3)
  • src/codex/catalog/provider-fetch.ts
  • tests/catalog-custom-model-metadata.test.ts
  • tests/catalog-vision-sidecar-modalities.test.ts

Comment on lines +16 to +17
modelReasoningEfforts: { "qwen-coder-3b": [] },
},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Cover the noReasoningModels regression path.

Line 16 sets modelReasoningEfforts to [], but this fixture does not declare noReasoningModels. A regression that stops inheriting noReasoningModels metadata can still pass.

Add noReasoningModels: ["qwen-coder-3b"] to this provider fixture. Keep the assertions for no supported reasoning levels and no default reasoning level.

Based on PR objectives: models in noReasoningModels with an empty effort list must expose neither generic reasoning levels nor a default reasoning level.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/catalog-custom-model-metadata.test.ts` around lines 16 - 17, Add
noReasoningModels containing "qwen-coder-3b" to the provider fixture alongside
modelReasoningEfforts in the relevant test setup. Preserve the existing
assertions that this model exposes no supported reasoning levels and no default
reasoning level, ensuring the regression path verifies inherited
noReasoningModels metadata.

@Wibias

Wibias commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Please put your Pull-Request on Ready for Review, once you are finished.

@lidge-jun

Copy link
Copy Markdown
Owner

Closing this in favour of #965, which fixes the same issue. Both landed within an hour of each other on #962 — that is a collision, not a fault, and the diff itself is clean work.

Why #965 won, concretely. #962 is about a custom row replacing a same-slug provider-derived row. #965 inherits from the row actually being replaced, so it also retains live /models metadata such as normalized capabilities. This PR instead recomputes catalogHintsFromProviderConfig() for every custom row (src/codex/catalog/provider-fetch.ts:785-809), including custom-only rows that never had a provider counterpart — broader than #962 requires, and it cannot retain discovered metadata because it rebuilds from config rather than inheriting.

The part that decided it: this PR changes tests/catalog-vision-sidecar-modalities.test.ts:102-126 from asserting "no registry reasoning metadata leaks onto an unmatched custom override" to expecting that leak, and removes three fetch should not be called guards. That assertion encodes a deliberate earlier decision from the #349/#344 vision-sidecar work. Rewriting it to accommodate a broader change is the kind of edit that should be its own proposal with its own argument, not a step inside a bug fix.

Verified rather than assumed — both branches were checked out and ablated. #965's ablation fails exactly one test, the #962 regression. This PR's ablation fails two, one of which is the rewritten contract above.

Your implementation does handle noReasoningModels correctly, and if you think the unmatched-custom-row behaviour should change, that is a real question worth raising on its own — I would read that issue.

#965 is carried in #973 (stack 6/6). Thanks for the fix, and sorry it was the duplicate.

@lidge-jun lidge-jun closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants