Rank Content search results by relevance

[3mdistal]

Problem

Content search previously ordered every matching document by modification time. A recent document that mentioned the query incidentally in its body could therefore appear above the document whose title was the intended match. For example, searching task prio ranked an article containing the words task and prior above Task Priorities.

That behavior made the picker feel arbitrary and forced people to scan through incidental matches even when a strong title match existed.

Approach

Rank the complete authorized result set with protected relevance tiers before applying pagination. Exact titles rank first, followed by contiguous title prefixes, title word-prefix matches, title substring matches, description matches, and body-only matches. Modification time remains the tie-breaker within a tier, so a recent body match cannot overtake a title match.

Body-only results receive a small phrase-coherence signal on result sets of at most 1,000 candidates. Title tier evaluation short-circuits expensive checks for clear body-only matches. Larger result sets retain the protected title and description tiers while skipping that full-body tie-breaker to keep latency predictable. Snippets prefer a passage containing more positive query terms while retaining bounded fallbacks for reverse-order, widely separated, whitespace-normalized, or pathological matches. Search eligibility, access checks, exclusions, filters, and the existing query language remain unchanged.

What changed

• Added a shared SQL ranking helper for title tiers, field coverage, and coherent body phrases.
• Applied ranking before limit and offset, with a window count for ordinary pages and the existing explicit count fallback for offsets past the final page.
• Made OR alternatives symmetric in exact and prefix title tiers, kept excluded description/body fields neutral during title-only search, and prevented intitle: terms from influencing description ranking.
• Hydrated bounded snippets in a second query over the selected page so full-set ranking does not evaluate snippet expressions for every candidate.
• Capped snippet occurrence discovery at 256 candidates per request, projected preview text at 5,000 characters, and final snippets at their existing display bound.
• Added database-backed adversarial, lifecycle, pagination, stability, mutation, access-control, OR-symmetry, field-scope, whitespace, pathological-query, and performance fixtures.
• Updated Content's search capability documentation, agent guidance, and changelog.

Safety and operations

This changes read ordering and snippet selection only. It adds no migration, data write, credential, external service, or rollout dependency. The query remains access-scoped before ranking. The main performance risk is evaluating relevance over many eligible rows, so the suite includes mixed-size documents and a 10,000-document warm latency gate.

Verification

• Focused database and adversarial suites pass: 31 tests covering title tiers, OR symmetry, title-only field isolation, stable equal-rank ordering, access and trash rules, mutation visibility, bounded oversized phrases, whitespace-normalized phrases, and pagination integrity.
• The standalone 10,000-document fixture passes at 253.0 ms warm p95 on the final head against a 400 ms budget; a concurrent focused run measured 323.3 ms. A 200 KB repeated-token document and mixed-size rare/common/zero-result fixtures also remain within their budgets.
• Content typecheck passes. Relevant query-scope, silent-coercion, blob-predicate, route, migration, localization, and product-conformance guards pass.
• Live picker QA with a paginated 24-result fixture shows Task Priorities first for task prio, ahead of newer body-only matches; the next page contains the remaining results without duplication.
• Independent review identified deep-snippet fallback, performance, OR symmetry, title-only field leakage, recursive scan growth, whitespace normalization, and oversized-phrase risks. The fixes and regressions are included in this head.

Review focus

• Do the protected title tiers match expected user intent without changing which documents are eligible?
• Does the phrase-coherence signal remain appropriately weak and bounded relative to title and description matches?
• Are the request-wide occurrence cap, 5,000-character projection cap, and fallback behavior safe for pathological bodies and queries?
• Is the count(*) over() plus empty-page fallback correct for all pagination boundaries?

[github-actions]

Here's a visual recap of what changed:

Open the full interactive recap

[3mdistal]

Addressed the three review findings in df18d51226:

• Snippet hydration now runs in a second query restricted to the selected page, so the full authorized candidate set pays ranking cost but not recursive snippet extraction. The 10,000-document gate passes locally at 377.5 ms warm p95 against the 400 ms budget.
• OR groups already bypass the single-query exact/prefix tier when they contain multiple alternatives. I added a regression that searches both amber OR violet and violet OR amber and verifies identical ordering, making that symmetry explicit.
• Match-window projection now has a hard 5,000-character cap. A new oversized quoted-phrase regression verifies the returned snippet remains bounded.

Focused ranking, discovery, and adversarial suites pass (29 tests), and Content typecheck passes on the updated head.

• Codex AI

[3mdistal]

Addressed the incremental review findings in 89be19623d:

• Title-only searches now exclude description coverage and body proximity from ranking, with a regression proving excluded fields cannot alter order.
• Exact and prefix title tiers now score every alternative in a single OR group symmetrically; both operand orders have regression coverage.
• Recursive occurrence discovery now shares a 256-candidate budget across the entire request rather than applying 256 per term.
• Snippet matching normalizes query whitespace alongside body whitespace, so valid quoted phrases still anchor the preview.
• Body phrase coherence is now limited to result sets of at most 1,000 candidates, while protected title/description tiers continue to rank the full authorized result set.

The focused ranking/discovery matrix passes all 31 tests. The standalone 10,000-document fixture passed twice at 264.7 ms and 253.9 ms warm p95 against its 400 ms budget. Content typecheck also passes.

• Codex AI

[builder-io-integration]

Builder reviewed your changes and found 2 potential issues 🟡

Review Details

Incremental Code Review Summary

The latest commit fixes all four previously open findings: OR alternatives now receive symmetric exact/prefix tiers, title-only searches disable non-title ranking signals, snippet occurrence expansion has a 256-total budget, and quoted snippets normalize whitespace consistently. Those four stale review threads were resolved before this review.

New Findings

• 🟡 MEDIUM — The 10k-document performance contract has regressed again: the ranking query measured approximately 673 ms warm p95 against the asserted 400 ms budget in one independent run. The full authorized set is still evaluated before pagination, so this remains a real request-path latency issue rather than an unused code path.
• 🟡 MEDIUM — intitle: terms still contribute to description coverage and can reorder results based on a field that the operator explicitly excludes. Title-only operator terms should remain neutral outside title ranking.

The two-query page hydration and access-scoped preview predicate are sound, and the focused functional suites reported by agents passed. Standard risk remains appropriate.

🧪 Browser testing: Could not complete — the browser-test-planner invocation was aborted after the dev server was confirmed healthy, so no visual evidence is available for this incremental review.

[builder-io-integration]

Builder reviewed your changes and has a few items to flag 🟡

Review Details

Incremental Code Review Summary

The latest commit addresses the previously open intitle: ranking leakage: title-only terms are now excluded from description ranking, and that stale thread was resolved. The request-wide snippet occurrence cap and ranking short-circuit changes are directionally sound, and the focused functional suites reported by reviewers pass across ranking, operators, pagination, snippets, lifecycle, and access behavior.

The existing 400 ms warm-search performance comment remains unresolved and was intentionally not reposted. Multiple independent runs still report the 10k-document benchmark above the committed budget, so the PR does not yet satisfy its stated performance target.

No additional confirmed bugs were found in this incremental pass.

🧪 Browser testing: Could not complete — the dev server was healthy, but browser-test executors still have no mcp__chrome__* tools, so the 15-case visual plan could not produce UI evidence.