Add negative constraint parsing to lexical search

[Mbeaulne]

Description

Adds support for negative constraints in lexical search queries. When a user includes phrases like "not GCS", "excluding GCS", or "without GCS" in their search, components matching those excluded terms are filtered out of the results. This allows users to express intent more naturally, such as "I want to upload a file but not to GCS", and receive only the relevant components.

This is implemented by parsing the query text before tokenization, extracting negative constraint phrases using a regex pattern, and scoring any index entry that matches a negative token as zero. The word "but" has also been added to the stop words list to avoid it interfering with scoring.

Related Issue and Pull requests

Type of Change

• Bug fix
• New feature
• Improvement
• Cleanup/Refactor
• Breaking change
• Documentation update

Checklist

• I have tested this does not break current pipelines / runs functionality
• I have tested the changes on staging

Screenshots (if applicable)

Test Instructions

1. Build the search index with components that have overlapping keywords (e.g., two upload components, one for local and one for GCS).
2. Run a search query containing a negative constraint such as "upload a file but not to GCS" or "upload a file excluding GCS".
3. Verify that only the component not matching the excluded term is returned.

Additional Comments

The negative constraint pattern currently recognises the trigger words without, excluding, exclude, not, and no, optionally followed by prepositions like to, use, or using.

[github-actions]

🎩 Preview

A preview build has been created at: 06-18-parse_negative_constraints_without_not_no_exclude_/3b7ef74

[Mbeaulne]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Simplify component search result cards #2459
• Add context-aware component search suggestions #2458
• Add component discovery docs #2457
• Add component search analytics baseline #2456
• Limit dashboard component browse rendering #2454
• Add component search empty state suggestions #2453
• Decouple editor component search input #2452
• Polish component search explanations #2451
• Add compatible component suggestions #2450
• Add component collection search results #2449
• Add component lifecycle badges #2448
• Add component search URL state #2447
• Disable AI search when literal search finds no matches #2444
• Debounce component search input #2433
• add client-side embeddings cached in IndexedDB #2432
• Add search quality expectation tests for lexical search #2431
• Add deep AI search to rerank all components in selected sources #2430
• Add source-diverse AI rerank candidate pool #2429
• Add negative constraint parsing to lexical search #2428 👈 (View in Graphite)
• Add fuzzy/typo-tolerant matching for component name and I/O fields #2427
• Improve component search scoring relevance #2426
• Add synonym expansion to component lexical search #2425
• Normalize component search tokens for better matching #2424
• Expand component search indexing fields #2423
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[camielvs]

🤖 Code review — Add negative constraint parsing to lexical search

Nicely thought-through. The two design decisions that matter most are both correct and well-tested: negation is literal (negated terms are not synonym-expanded, so "not gcs" doesn't also drop storage/bucket components), and exclusion is a whole-token match rather than substring (a short negated token can't knock a component out by appearing inside an unrelated word). The conjunction-barrier in the capture group (not gcs and train → exclude only gcs, keep train) is a genuinely subtle case and the regex + comment handle it well. Zero added cost when there's no negation (empty negativeTokens short-circuits).

The main thing to weigh:

• Hard exclusion across all fields is more aggressive than the AI layer's negation, and it happens upstream of it. scoreEntry returns score: 0 if a negated token matches any field — including description, implementation, and metadata. Because the AI rerank candidate pool is built from lexicalSearch results, an excluded component never reaches the reranker at all. So a multi-cloud uploader whose description reads "uploads to GCS, S3, and Azure" is silently dropped by "upload not gcs" — even though it genuinely satisfies "upload, but not only GCS". Keyword negation can't distinguish "uses X" from "mentions X", and the LLM (which already has negative-constraint rules in its system prompt) handles this far more nuancedly. Two options worth considering:

  • Limit hard negative exclusion to name + io (mirroring the precision-field choice you made for fuzzy matching in Add fuzzy/typo-tolerant matching for component name and I/O fields #2427), so incidental description/metadata mentions don't trigger a drop; or
  • Down-weight rather than hard-drop, and let the LLM make the final call.

• Stemmed negation can over-exclude on common words. negativeTokens come from baseSearchTokens (stemmed), so "no models" excludes both models and model. Usually desirable, but model/data/input appear across many components' io/description, so a negated common term can wipe out a large slice. This is inherent to keyword negation on NL phrasing — mostly a "be aware" note, and limiting exclusion to name/io (above) would also blunt it.

Regex looks safe from catastrophic backtracking (word groups separated by required \s+, no ambiguous nested quantifiers), and the excluding|exclude ordering correctly prefers the longer alternative. Tests cover the literal-vs-synonym distinction well — a negative test for "not gcs" against a description-only GCS mention would be a good addition to pin down whichever field-scope decision you land on.