[doc](inverted-index) Document the norms index property and its VARIANT default by eldenmoon · Pull Request #4146 · apache/doris-website

eldenmoon · 2026-09-16T05:01:08Z

Documents the norms inverted index property added by apache/doris#68039.

docs/table-design/index/inverted-index/scoring.md: where |d| comes from, the one byte per row per indexed field cost, the per-index norms property with its defaults, and what scoring does when an index has no norms.
docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md: a tokenized index on a VARIANT path writes no norms by default, the reason (one index per path, so rows x paths bytes), the effect on score(), and how to opt back in with "norms" = "true".

Chinese pages updated alongside. Dev docs only: the property exists on master, so versioned_docs/version-4.x and older are intentionally untouched.

…NT default The scoring page now explains where |d| comes from, that norms cost one byte per row for every indexed field, and how the norms property overrides the default. The VARIANT page states that a tokenized index on a VARIANT path writes no norms by default, why (one index per path, rows x paths bytes), what that means for score(), and how to opt back in with "norms" = "true". Documents apache/doris#68039. Dev docs only: the property exists on master, so versioned 4.x and older are left unchanged.

…onfig The norms index property defaults to true on every index. An index on a VARIANT path leaves norms out because the BE config inverted_index_skip_norms_for_variant (default true, changeable at runtime) skips them there, not because the property itself defaults to false on such an index. The property still decides on its own in either direction, and a subpath index inherits it from the index it comes from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

An analyzed index writes BM25 norms on a VARIANT path just as it does on an ordinary column. The BE config inverted_index_skip_norms_for_variant is off by default and, once turned on, drops norms for every index on a VARIANT path whatever that index's norms property says. Both examples now set "norms" = "false", the only direction the property is still useful in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

### What problem does this PR solve? Problem Summary: The "norms" property did not mean the same thing on every index. An analyzed index on a variant path defaulted to writing no norms while an analyzed index on any other column defaulted to writing them, so the default came from the shape of the index and the property only corrected it. That also changed how variant path indexes rank, for every user, as soon as the fix landed. Give the property one meaning everywhere -- an analyzed index writes norms unless the property says otherwise, on a variant path as on an ordinary column -- and move the variant skip into the new mutable BE config inverted_index_skip_norms_for_variant, which is off by default. Turning it on drops norms for every index on a variant path (an index declared with a field_pattern, or the copy that each extracted subcolumn inherits from it) whatever that index's property says, so a cluster that cares about the space can reclaim it without rewriting its index definitions, and a cluster that cares about ranking keeps what it has. Default behaviour therefore matches the state before this PR: norms are written everywhere. ### Release note Added the BE config inverted_index_skip_norms_for_variant (off by default, mutable). Turning it on drops BM25 norms for every inverted index on a VARIANT path, whatever the index's "norms" property says. ### Check List (For Author) - Test: Unit Test / Regression test - be/test/storage/segment/inverted_index_writer_test.cpp: NormsFollowIndexNormsProperty covers the default and the property in both directions on an ordinary column, a variant subcolumn and a field_pattern index, and then the config dropping norms for variant path indexes (including one that sets "norms" = "true") while leaving an ordinary column index alone (InvertedIndexWriterTest.*: 18 passed) - regression-test/suites/inverted_index_p0/test_variant_subcolumn_index_norms.groovy: the default and "norms" = "false" on field_pattern and whole-column VARIANT indexes, plus a setBeConfigTemporary case proving the config takes effect without a restart; the .out was regenerated because variant paths now keep norms, so BM25 length normalization changes those scores - test_bm25_score_variant and test_omit_norms rerun on the same build - Behavior changed: No (defaults unchanged; the variant skip is opt-in) - Does this need documentation: Yes (apache/doris-website#4146) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…s without norms ### What problem does this PR solve? Problem Summary: The "norms" index property and inverted_index_skip_norms_for_variant only reached the CLucene writer. SNII decided on its own: its writer set _writes_norms from the analyzer and the positions flag, and its direct compaction from the analyzer alone, so on a SNII table both knobs were silently ignored. SNII scoring also rejected any segment without norms, so it could not have honoured them anyway. - Move the rule into one function, should_write_index_norms(const TabletIndex&), and call it from the CLucene writer, the SNII writer and SNII direct compaction. SNII direct compaction rebuilds norms from the merged postings, so sources with and without norms merge correctly; the other SNII compaction path goes through the SNII writer. - Let SNII score a segment without norms. SNII keeps the token count in its stats block, so avgdl stays correct. SniiStatsProvider::encoded_norm now returns std::nullopt for such a segment, and ScorerContext::score scores that document as if its length were avgdl, the same length-neutral form the CLucene BM25 fix uses. A document with a norm goes through the same arithmetic as before. resolve_snii_scoring_segment no longer takes has_norms. - Mark both norms regression suites nonConcurrent, since they flip a BE config. ### Release note The "norms" inverted index property and the BE config inverted_index_skip_norms_for_variant now also apply to the SNII index storage format, and SNII scoring no longer fails on a segment written without norms. ### Check List (For Author) - Test: Unit Test / Regression test - SniiWriterNorms.WritesNormsFollowSharedNormsPolicy, SniiCompactionEligibilityTest.DestinationWritesNormsFollowSharedNormsPolicy, SniiScoringQuery.IndexWithoutNormsScoresWithoutLengthNormalization, CollectionStatisticsTest.SniiScoringAdmitsSegmentWithoutNorms - regression inverted_index_p0/storage_format/test_storage_format_snii_norms (new), plus test_variant_subcolumn_index_norms, test_bm25_score_variant, test_omit_norms and test_storage_format_snii - Behavior changed: Yes. A SNII segment without norms is now scored without length normalization instead of failing the query. - Does this need documentation: Yes (apache/doris-website#4146) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ithout norms score A row without norms is scored as if its length were avgdl, so the whole length factor 1 - b + b * |d| / avgdl becomes 1, rather than one term of it dropping out. Turning inverted_index_skip_norms_for_variant on, or setting "norms" = "false", should wait until every BE is upgraded, because an older BE cannot score an index written without norms. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

### What problem does this PR solve? Problem Summary: The previous commits of this PR let an analyzed index be written without norms ("norms" = "false", or inverted_index_skip_norms_for_variant on a variant path), and scored such an index without length normalization. That left two problems: - CLucene keeps a field's token count in its .nrm header. A segment without norms adds no tokens but still adds documents, and its rows read the fake norm encodeNorm(0), i.e. length 0. When some segments of a field have norms and others do not, as while the config is being turned on, avgdl stays positive, the zero-avgdl fallback never runs, and the rows without norms rank as the shortest possible documents, above otherwise identical rows. - SNII was changed to score such segments too, which made the two formats and the transition harder to reason about. Follow what SNII already did: BM25 scoring needs norms from every segment it reads. - CollectionStatistics now refuses an analyzed CLucene segment whose field has no norms with INVERTED_INDEX_NOT_SUPPORTED, before any score is computed, so a collection with some or all segments lacking norms fails the score() query instead of ranking inconsistently. An index that is not analyzed never writes norms and is collected as before, so SEARCH scoring over keyword fields is unchanged. - The zero-avgdl fallback in BM25Similarity is dropped: with the refusal it can no longer be reached by an analyzed index. - SNII scoring is back to rejecting segments without norms (resolve_snii_scoring_segment, SniiStatsProvider::encoded_norm, ScorerContext::score are as on master). Its message now names the property and the config instead of suggesting that compaction will add the norms back. - The writer side is unchanged: should_write_index_norms still decides for the CLucene writer, the SNII writer and SNII direct compaction. - Fix the config comment, which still said the property could override the config. MATCH filtering is not affected; only queries that compute score() need norms. ### Release note A query that computes score() on an analyzed inverted index now fails when a segment it reads was written without norms ("norms" = "false", inverted_index_skip_norms_for_variant, or a segment written before norms were supported), instead of returning NaN or inconsistent scores. ### Check List (For Author) - Test: Unit Test / Regression test - CollectionStatisticsTest.LegacyV3RejectsSegmentWrittenWithoutNorms (alone and mixed with a segment that has norms), CollectionStatisticsTest.LegacyV3KeywordIndexWithoutNormsIsStillCollected - regression test_variant_subcolumn_index_norms and test_storage_format_snii_norms: MATCH still filters, score() fails on indexes without norms and on a mixed-generation table; plus test_bm25_score_variant, test_omit_norms, test_search_score_topn_predicates, test_search_score_cache, test_search_score_topn_delete_predicate and test_storage_format_snii - Behavior changed: Yes. score() on an analyzed index with a segment that has no norms fails instead of returning NaN or skewed scores. - Does this need documentation: Yes (apache/doris-website#4146) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A query that computes score() on a tokenized index now fails when any segment it reads has no norms for the field, including a table where only some segments were written without them, while MATCH filtering is unaffected. Describe that, and present "norms" = "false" and inverted_index_skip_norms_for_variant as options for indexes that are never ranked with score(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…kip norms on variant paths (#68039) ### What problem does this PR solve? Problem Summary: An inverted index on a VARIANT column is copied to every extracted subcolumn, so a segment holds one copy of it per path. Since #53980 every analyzed index calls `setOmitNorms(false)`, and CLucene writes norms densely: 12 header bytes plus **one byte per segment row for every indexed field**, including the rows that hold no value for that field. For a VARIANT column that cost is multiplied by the number of paths. On a real table (~1400 paths, 7 non-BKD indexes per path) a single base-compacted segment held 9891 `.nrm` files of 4,055,884 bytes each: **37.36 GiB of norms out of a 38.47 GiB `.idx`, 97.1%**. Segments written before the upgrade had no `.nrm` at all, and the non-norms bytes had not grown. Reproduced end to end on 1M rows / 1000 paths / 11 `field_pattern` indexes, by loading with 3.1 and then upgrading the same data in place to 4.1: | | index size after full compaction | `.nrm` | | --- | --- | --- | | 3.1 | 99.6 MiB | none | | 4.1, same data | 934.1 MiB | 875 files x 1,000,012 B = 834.5 MiB | **9.4x**, and the norms are the whole difference. A/B on the same data: norms on 457.3 MiB vs norms off 99.6 MiB (**4.6x**), with identical `MATCH` row counts and sums. #### What this PR changes This PR gives operators a way to reclaim that space without changing what anyone gets by default, and applies it the same way to the CLucene formats (V2/V3) and to SNII. 1. **A `norms` index property.** An analyzed index writes norms unless it sets `"norms" = "false"`, on a VARIANT path exactly as on an ordinary column. A subcolumn copy inherits the properties of the index it came from, so the property on a whole-column VARIANT index also settles every one of its subcolumn copies. 2. **A mutable BE config `inverted_index_skip_norms_for_variant`, off by default.** Turning it on drops norms for every index on a VARIANT path (an index declared with a `field_pattern`, or the copy an extracted subcolumn inherits, which carries the path as its index suffix), whatever that index's property says, so a cluster hit by the size regression can recover the space without rewriting its index definitions. 3. **One policy function.** `should_write_index_norms(const TabletIndex&)` holds the rule, and the three places that decide whether norms get written all call it: the CLucene writer, the SNII writer, and SNII direct compaction. SNII direct compaction rebuilds norms from the merged postings rather than copying them; the other SNII compaction path rebuilds the index through the SNII writer. 4. **BM25 scoring needs norms from every segment it reads, in both formats.** `MATCH` filtering is unaffected; a query that computes `score()` on an analyzed index fails with `INVERTED_INDEX_NOT_SUPPORTED` when any segment of the field has no norms. - SNII already worked this way (`resolve_snii_scoring_segment`); its message now names the property and the config instead of suggesting that compaction will bring the norms back. - CLucene did not. It keeps a field's token count in the `.nrm` header, and a row without norms reads the fake norm `encodeNorm(0)`, i.e. length 0. With no segment carrying norms, `avgdl` was 0 and `score()` came out NaN. With only some segments carrying norms, as while the config is being turned on, `avgdl` stayed positive and the rows without norms ranked as the shortest possible documents, above otherwise identical rows. `CollectionStatistics` now refuses such a segment before any score is computed. An index that is not analyzed never writes norms and is collected as before, so `SEARCH` scoring over keyword fields is unchanged. #### Compatibility - Defaults are unchanged: norms are still written everywhere unless someone opts out, and an index that has norms scores exactly as before. - No storage format change. - Turning the config on, or setting `"norms" = "false"`, is for indexes that are never ranked with `score()`. For such an index, `score()` fails as soon as one segment without norms exists. - A table upgraded from a version that did not write norms (before 4.0) now fails `score()` on its analyzed indexes until compaction has rewritten the old segments, where it used to return NaN or skewed scores. - Change the property or the config only after every BE runs this version. ### Release note Added the `norms` inverted index property and the BE config `inverted_index_skip_norms_for_variant` (off by default, mutable), which drops BM25 norms for inverted indexes on VARIANT paths and can cut the index size of a VARIANT column with many indexed paths by several times. Both apply to the V2, V3 and SNII index storage formats. A query that computes `score()` on an analyzed index now fails when a segment it reads has no norms, instead of returning NaN or inconsistent scores. ### Check List (For Author) - Test: Unit Test / Regression test - `be/test/storage/segment/inverted_index_writer_test.cpp`: `NormsFollowIndexNormsProperty` - `be/test/storage/index/snii_writer_test.cpp`: `SniiWriterNorms.WritesNormsFollowSharedNormsPolicy` - `be/test/storage/index/snii/compaction/snii_compaction_eligibility_test.cpp`: `DestinationWritesNormsFollowSharedNormsPolicy` - `be/test/storage/index/inverted/similarity/collection_statistics_test.cpp`: `LegacyV3RejectsSegmentWrittenWithoutNorms` (alone and next to a segment with norms), `LegacyV3KeywordIndexWithoutNormsIsStillCollected`, and the existing SNII rejection tests - `fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java` - Regression: `inverted_index_p0/test_variant_subcolumn_index_norms` (checks the `.nrm` files) and `inverted_index_p0/storage_format/test_storage_format_snii_norms` (reads norms off the scores): `MATCH` still filters, `score()` fails without norms, including on a mixed-generation table. Both are nonConcurrent, since they flip the BE config. Also rerun: `test_bm25_score_variant`, `test_omit_norms`, `test_search_score_topn_predicates`, `test_search_score_cache`, `test_search_score_topn_delete_predicate`, `test_storage_format_snii` - Behavior changed: No for defaults. `score()` on an analyzed index with a segment that has no norms now fails instead of returning NaN or skewed scores. - Does this need documentation: Yes (apache/doris-website#4146)

eldenmoon mentioned this pull request Sep 16, 2026

[fix](inverted index) Add a norms index property and a BE config to skip norms on variant paths apache/doris#68039

Merged

eldenmoon deployed to Production September 16, 2026 05:01 — with GitHub Actions Active

eldenmoon deployed to Production September 16, 2026 07:32 — with GitHub Actions Active

eldenmoon deployed to Production September 16, 2026 07:48 — with GitHub Actions Active

eldenmoon deployed to Production September 16, 2026 10:10 — with GitHub Actions Active

eldenmoon deployed to Production September 16, 2026 13:28 — with GitHub Actions Active

Navigation Menu

[doc](inverted-index) Document the norms index property and its VARIANT default - #4146

[doc](inverted-index) Document the norms index property and its VARIANT default#4146
eldenmoon wants to merge 5 commits into
apache:masterfrom
eldenmoon:docs/inverted-index-norms

eldenmoon commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

eldenmoon commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant