branch-4.1: [fix](inverted index) Add a norms index property and a BE config to skip norms on variant paths by eldenmoon · Pull Request #68140 · apache/doris

eldenmoon · 2026-09-17T13:29:26Z

What problem does this PR solve?

Issue Number: None

Related PR: #68039

Problem Summary:

This is a selective backport of #68039 to branch-4.1. It keeps the CLucene/V2/V3 VARIANT-path
norms handling and deliberately excludes all SNII-related changes.

An inverted index on a VARIANT column is copied to every extracted subcolumn, so dense BM25 norms
can consume substantial storage when many paths are indexed. This backport adds the norms index
property and the mutable BE config inverted_index_skip_norms_for_variant (off by default). The
config omits norms for VARIANT-path indexes regardless of the index property.

CLucene BM25 scoring now rejects an analyzed index when any segment was written without norms,
while MATCH filtering remains available. This prevents NaN or misleading scores for mixed-generation
segments. Whole-column and subcolumn VARIANT regression coverage is included.

SNII scope:

The upstream SNII policy/compaction changes are not included.
No files under be/src/storage/index/snii/** or SNII-specific tests/regressions are changed.
The final diff contains no SNII-related paths or additions.

Release note

Added the norms inverted-index property and the BE config inverted_index_skip_norms_for_variant
for CLucene/V2/V3 VARIANT-path indexes. BM25 score() now fails when an analyzed segment has no
norms instead of returning invalid or inconsistent scores.

Check List (For Author)

Test:
- ./build.sh --fe: BUILD SUCCESS; Checkstyle reports 0 violations.
- ./run-fe-ut.sh --run org.apache.doris.analysis.InvertedIndexNormsPropertyTest: 2 passed.
- clang-format check and git diff --check: passed.
- The 4.1 regression setup no longer sets the unsupported enable_segment_limit_pushdown
  session variable. The target regression was not executed locally because the BE build is
  blocked by the available Lance-C header/library mismatch (lance_session_manager.cpp
  expects session APIs absent from thirdparty/installed/include/lance/lance.h). The ASAN
  configuration is additionally blocked by LeakSanitizer under ptrace/OpenBLAS getarch;
  the regression golden was reused from the upstream non-SNII result and was not regenerated
  locally.
Behavior changed: Yes (BM25 scoring is rejected for analyzed indexes without norms; defaults
remain unchanged).
Does this need documentation: Yes (follow-up to the upstream documentation work).

### What problem does this PR solve? Issue Number: None Related PR: apache#68039 Problem Summary: Analyzed inverted indexes inherited by sparse VARIANT subcolumns write dense norms. Add the variant norms switch and skip norms for those indexes by default while retaining norms for ordinary analyzed indexes and an opt-in compatibility setting. ### Release note Variant subcolumn inverted indexes no longer write BM25 norms by default. ### Check List (For Author) - Test: Source-level verification only in this commit; focused build/test follows. - Regression test / Unit Test / Manual test / No need to test (with reason) - Behavior changed: Yes (variant subcolumn norms default to omitted) - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#68039 Problem Summary: Make norms an inverted-index property. Variant path indexes omit norms by default, while the norms property can restore them per index; ordinary column indexes keep norms by default. ### Release note Inverted indexes accept the norms property. ### Check List (For Author) - Test: Pending focused build and test verification - Behavior changed: Yes (norms are controlled per index) - Does this need documentation: Yes

### What problem does this PR solve? Issue Number: None Related PR: apache#68039 Problem Summary: Extend the norms regression coverage to whole-column VARIANT indexes and verify that per-subcolumn copies inherit the default or explicit norms property. ### Release note None ### Check List (For Author) - Test: Regression test added; execution pending - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#68039 Problem Summary: Add an opt-in BE config that omits BM25 norms for indexes on variant paths, regardless of their per-index norms property, while preserving ordinary-column behavior. ### Release note Add inverted_index_skip_norms_for_variant for reducing dense norms on variant paths. ### Check List (For Author) - Test: Focused unit and regression coverage added; execution pending - Behavior changed: Yes (opt-in BE config) - Does this need documentation: Yes

### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: apache#68039\n\nProblem Summary: BM25 scoring used a zero average document length when a CLucene segment omitted norms, producing invalid or misleading scores. Reject scoring when an analyzed index has a segment without norms, while keeping MATCH filtering available and preserving the VARIANT skip-norms configuration behavior. This selective backport excludes all SNII changes.\n\n### Release note\n\nBM25 scoring now reports an error for analyzed indexes whose segments were written without norms.\n\n### Check List (For Author)\n\n- Test: FE build, FE unit test, format check; BE build and regression are blocked by the local ASAN/toolchain environment.\n- Behavior changed: Yes (BM25 scoring without norms is rejected)\n- Does this need documentation: No

hello-stephen · 2026-09-17T13:29:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

eldenmoon · 2026-09-17T14:15:02Z

run buildall

hello-stephen · 2026-09-17T16:45:36Z

FE UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-17T19:53:12Z

BE UT Coverage Report

Increment line coverage 56.00% (14/25) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.90% (26026/43447)
Line Coverage	44.68% (269223/602503)
Region Coverage	40.49% (213460/527182)
Branch Coverage	41.98% (98681/235081)

hello-stephen · 2026-09-17T20:48:49Z

BE Regression && UT Coverage Report

Increment line coverage 56.00% (14/25) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	62.99% (26626/42271)
Line Coverage	47.54% (284175/597712)
Region Coverage	43.67% (230760/528379)
Branch Coverage	44.68% (104928/234821)

eldenmoon · 2026-09-18T02:42:27Z

run buildall

hello-stephen · 2026-09-18T04:32:42Z

BE UT Coverage Report

Increment line coverage 56.00% (14/25) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.90% (26026/43447)
Line Coverage	44.69% (269242/602503)
Region Coverage	40.55% (213790/527182)
Branch Coverage	41.99% (98721/235081)

hello-stephen · 2026-09-18T06:12:21Z

BE Regression && UT Coverage Report

Increment line coverage 72.00% (18/25) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.48% (31482/42271)
Line Coverage	58.80% (351474/597712)
Region Coverage	55.59% (293706/528379)
Branch Coverage	56.38% (132381/234821)

eldenmoon added 5 commits September 17, 2026 21:18

eldenmoon requested a review from yiguolei as a code owner September 17, 2026 13:29

[fix](regression) Remove unsupported session variable

28d31cb

yiguolei approved these changes Sep 19, 2026

View reviewed changes

eldenmoon merged commit 851858d into apache:branch-4.1 Sep 19, 2026
33 of 36 checks passed

eldenmoon mentioned this pull request Sep 20, 2026

branch-4.1: [fix](regression) Enable expression pushdown for variant norms test #68230

Merged

Navigation Menu

branch-4.1: [fix](inverted index) Add a norms index property and a BE config to skip norms on variant paths - #68140

branch-4.1: [fix](inverted index) Add a norms index property and a BE config to skip norms on variant paths#68140
eldenmoon merged 6 commits into
apache:branch-4.1from
eldenmoon:branch-4.1-pick-68039-no-snii

eldenmoon commented Sep 17, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Sep 17, 2026

Uh oh!

eldenmoon commented Sep 17, 2026

Uh oh!

hello-stephen commented Sep 17, 2026

Uh oh!

hello-stephen commented Sep 17, 2026

Uh oh!

hello-stephen commented Sep 17, 2026

Uh oh!

eldenmoon commented Sep 18, 2026

Uh oh!

hello-stephen commented Sep 18, 2026

Uh oh!

hello-stephen commented Sep 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

eldenmoon commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Sep 17, 2026

Uh oh!

eldenmoon commented Sep 17, 2026

Uh oh!

hello-stephen commented Sep 17, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 17, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 17, 2026

BE Regression && UT Coverage Report

Uh oh!

eldenmoon commented Sep 18, 2026

Uh oh!

hello-stephen commented Sep 18, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 18, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eldenmoon commented Sep 17, 2026 •

edited

Loading