Add deep AI search to rerank all components in selected sources

[Mbeaulne]

Description

Adds a Deep AI search button alongside the existing AI (smart) search button. While the standard AI search sends a limited, curated set of candidate components to the reranker, Deep AI search sends the entire searchable index — lexical hits first so truncating providers still see the most likely matches early — allowing the model to rerank across every available component in the selected sources.

Key changes:

• Introduces buildDeepAiCandidateMatches, which builds a candidate pool from all indexed components (no cap), ordered with lexical matches first and remaining components appended alphabetically.
• Exposes canDeepRerank and deepRerank from useComponentSearchV2State and wires them into both the Dashboard and Editor search UIs.
• Enriches RerankCandidate with richer I/O metadata (type and description, not just names) and a source field, giving the model more signal when ranking.
• Refactors the shared startAiSearch / startRerank helpers to accept an arbitrary candidate list, removing duplication between the standard and deep paths.
• Updates componentReferenceToCandidate to accept an optional source argument and serialize I/O types (including complex object types) via JSON.stringify.

Related Issue and Pull requests

Type of Change

• Bug fix
• New feature
• Improvement
• Cleanup/Refactor
• Breaking change
• Documentation update

Checklist

• I have tested this does not break current pipelines / runs functionality
• I have tested the changes on staging

Screenshots (if applicable)

Test Instructions

1. Open the component search panel (Dashboard or Editor).
2. Enter a non-empty query.
3. Confirm the Deep AI search button appears and is enabled when an AI provider is configured.
4. Click Deep AI search and verify that results are reranked across all components in the selected sources, not just the top lexical candidates.
5. Confirm the existing AI search (sparkles) button still behaves as before.
6. Verify the button is disabled when the query is empty or no AI provider is configured.

Additional Comments

scoreAllCandidates is intentionally set to false for deep rerank to avoid scoring overhead across the full index on every result row. The standard AI search retains scoreAllCandidates: true so relevance percentages continue to appear on displayed results.

[github-actions]

🎩 Preview

A preview build has been created at: 06-18-improve_ai_rerank_payload_for_component_search/49294fe

[Mbeaulne]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Simplify component search result cards #2459
• Add context-aware component search suggestions #2458
• Add component discovery docs #2457
• Add component search analytics baseline #2456
• Limit dashboard component browse rendering #2454
• Add component search empty state suggestions #2453
• Decouple editor component search input #2452
• Polish component search explanations #2451
• Add compatible component suggestions #2450
• Add component collection search results #2449
• Add component lifecycle badges #2448
• Add component search URL state #2447
• Disable AI search when literal search finds no matches #2444
• Debounce component search input #2433
• add client-side embeddings cached in IndexedDB #2432
• Add search quality expectation tests for lexical search #2431
• Add deep AI search to rerank all components in selected sources #2430 👈 (View in Graphite)
• Add source-diverse AI rerank candidate pool #2429
• Add negative constraint parsing to lexical search #2428
• Add fuzzy/typo-tolerant matching for component name and I/O fields #2427
• Improve component search scoring relevance #2426
• Add synonym expansion to component lexical search #2425
• Normalize component search tokens for better matching #2424
• Expand component search indexing fields #2423
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Mbeaulne]

🤖 This is an AI-generated code review comment.

[MEDIUM] src/hooks/useNaturalLanguageComponentSearch.ts ~lines 38-50 — The rerank mutation passes no AbortSignal, so a now-much-heavier deep rerank cannot be cancelled if the user retypes. Pre-existing, but materially amplified by this feature. Follow-up: thread signal into the mutation so a new query aborts the in-flight deep call.

(Posted as a PR-level comment because this file has no changed lines in the diff to anchor an inline comment to.)

[Mbeaulne]

🤖 This is an AI-generated response.

Re: the AbortSignal note on the rerank mutation (useNaturalLanguageComponentSearch.ts) — deferring to a follow-up rather than addressing it in this PR. It is genuinely cross-cutting (an AbortController in each caller, a signal field on RerankVariables, and rerankComponentsByNaturalLanguage forwarding it to fetch), it is pre-existing (the mutation never threaded a signal), and the deep-search cap (DEEP_AI_CANDIDATE_LIMIT) has already shrunk the cost of an un-aborted in-flight call. Tracking it as a separate cleanup so this PR stays scoped.