From a76fc4e992252e886f8e9b59910358aad4280588 Mon Sep 17 00:00:00 2001 From: eldenmoon Date: Wed, 16 Sep 2026 13:01:03 +0800 Subject: [PATCH 1/5] [doc](inverted-index) Document the norms index property and its VARIANT default The scoring page now explains where |d| comes from, that norms cost one byte per row for every indexed field, and how the norms property overrides the default. The VARIANT page states that a tokenized index on a VARIANT path writes no norms by default, why (one index per path, rows x paths bytes), what that means for score(), and how to opt back in with "norms" = "true". Documents apache/doris#68039. Dev docs only: the property exists on master, so versioned 4.x and older are left unchanged. --- .../sql-data-types/semi-structured/VARIANT.md | 19 +++++++++++++- .../index/inverted-index/scoring.md | 26 +++++++++++++++++++ .../sql-data-types/semi-structured/VARIANT.md | 19 +++++++++++++- .../index/inverted-index/scoring.md | 26 +++++++++++++++++++ 4 files changed, 88 insertions(+), 2 deletions(-) diff --git a/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md b/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md index 1c4fa1cda88a3..4e49970eb9c21 100644 --- a/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md +++ b/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md @@ -433,7 +433,7 @@ SELECT * FROM tbl WHERE v['id_2'] MATCH 'Apache'; In 3.1.x/4.0 and later, you can specify index properties for certain VARIANT subpaths, and even configure both tokenized and non-tokenized inverted indexes for the same path. Path-specific indexes require the path type to be declared via Schema Template. ```sql --- Common properties: field_pattern (target path), analyzer, parser, support_phrase, etc. +-- Common properties: field_pattern (target path), analyzer, parser, support_phrase, norms, etc. CREATE TABLE IF NOT EXISTS tbl ( k BIGINT, v VARIANT<'content' : STRING>, @@ -460,6 +460,23 @@ SELECT * FROM tbl WHERE v['pattern_1'] MATCH 'Doris'; SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris'; ``` +BM25 norms on VARIANT paths: + +A tokenized index on a VARIANT path does not write BM25 norms by default, while a tokenized index on an ordinary column does. Norms store the record length used by BM25 and cost one byte per row for every indexed field, written even for rows that hold no value for that path. One segment keeps one index per path, so norms on a VARIANT column with many paths would cost `rows × paths` bytes. Queries are unaffected apart from ranking: record-length normalization is skipped, and `score()` depends on term frequency and IDF only. + +Add `"norms" = "true"` to a path index when that path needs full BM25 scoring: + +```sql +CREATE TABLE IF NOT EXISTS tbl ( + k BIGINT, + v VARIANT<'title' : STRING, 'body_*' : STRING>, + -- title is ranked with record-length normalization + INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title", "norms" = "true"), + -- body_* keeps the default: no norms, no record-length normalization + INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*") +); +``` + Note: 2.1.7+ supports only InvertedIndex V2 properties (fewer files, lower write IOPS; suitable for disaggregated storage/compute). 2.1.8+ removes offline Build Index. ### When indexes don’t work diff --git a/docs/table-design/index/inverted-index/scoring.md b/docs/table-design/index/inverted-index/scoring.md index a74fae19b6613..2da24a9578bb8 100644 --- a/docs/table-design/index/inverted-index/scoring.md +++ b/docs/table-design/index/inverted-index/scoring.md @@ -111,6 +111,32 @@ Where: When a query contains multiple terms, **the final score is the sum of the scores of each term**. +### Record Length and the `norms` Property + +`|d|` comes from norms, which an index stores as one byte per row for every indexed field. The byte is written for every row of the segment, including rows that hold no value for that field, so an index that covers many sparse fields also pays for the rows it never matches. + +Norms are controlled per index with the `norms` property: + +| Index | Default | Norms written | +| -------------------------------------- | ------- | ------------- | +| Tokenized index on an ordinary column | `true` | Yes | +| Tokenized index on a VARIANT path | `false` | No | +| Non-tokenized index | - | Never | + +A VARIANT path index is an index declared with `field_pattern`, together with the copy of it that each extracted subpath inherits. One segment holds one such index per path, so writing norms there costs `rows × paths` bytes. + +Set the property explicitly to override the default: + +```sql +-- keep record-length normalization for one VARIANT path +INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title", "norms" = "true") + +-- drop it for an ordinary column +INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english", "norms" = "false") +``` + +When an index has no norms, scoring skips record-length normalization: the `b × |d| / avgdl` term drops out and the score depends on term frequency and IDF only. The property applies to newly written segments; segments written earlier keep their norms until compaction rewrites them. + ## Interpreting the Results Understanding the scoring results helps you use relevance ranking more accurately: diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md index e0b697d548a28..bfe3096f1e3c8 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md @@ -433,7 +433,7 @@ SELECT * FROM tbl WHERE v['id_2'] MATCH 'Apache'; 在 3.1.x/4.0 及之后的版本中,可为 VARIANT 的部分子列单独指定索引属性,甚至在同一路径上同时配置“分词与不分词”的两种倒排索引。指定 Path 索引需配合 Path 类型(Schema Template)使用。 ```sql --- 常用属性:field_pattern(目标子路径)、analyzer、parser、support_phrase 等 +-- 常用属性:field_pattern(目标子路径)、analyzer、parser、support_phrase、norms 等 CREATE TABLE IF NOT EXISTS tbl ( k BIGINT, v VARIANT<'content' : STRING>, @@ -460,6 +460,23 @@ SELECT * FROM tbl WHERE v['pattern_1'] MATCH 'Doris'; SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris'; ``` +VARIANT 路径上的 BM25 norms: + +VARIANT 路径上的分词索引默认不写 BM25 norms,而普通列上的分词索引会写。norms 保存 BM25 使用的记录长度,按行存储,每个被索引的字段每行 1 字节,即使该行在这个路径上没有值也会写入。一个段中每个路径各有一份索引,因此在路径数很多的 VARIANT 列上写 norms 的代价是 `行数 × 路径数` 字节。除排序外查询不受影响:仅跳过记录长度归一化,`score()` 只由词频和 IDF 决定。 + +如果某个路径需要完整的 BM25 打分,给该路径的索引加上 `"norms" = "true"`: + +```sql +CREATE TABLE IF NOT EXISTS tbl ( + k BIGINT, + v VARIANT<'title' : STRING, 'body_*' : STRING>, + -- title 参与记录长度归一化 + INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title", "norms" = "true"), + -- body_* 保持默认:不写 norms,不做记录长度归一化 + INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*") +); +``` + 注意:2.1.7+ 仅支持 InvertedIndex V2 属性(文件更少、写入 IOPS 更低,适配存算分离)。2.1.8+ 不再支持离线 Build Index 构建。 ### 索引失效问题 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md index 7d0dbcefeed18..67b1f19aae478 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md @@ -111,6 +111,32 @@ avgdl = total_terms / total_rows 当查询包含多个词项时,**最终得分为各词项得分之和**。 +### 记录长度与 `norms` 属性 + +公式中的 `|d|` 来自 norms。索引会为每个被索引的字段按行存储 norms,每行 1 字节。这个字节对段内每一行都会写入,包括该字段没有值的行,因此覆盖大量稀疏字段的索引也要为从不命中的行付出空间。 + +norms 由索引属性 `norms` 控制: + +| 索引 | 默认值 | 是否写 norms | +| ------------------------ | ------- | ------------ | +| 普通列上的分词索引 | `true` | 写 | +| VARIANT 路径上的分词索引 | `false` | 不写 | +| 非分词索引 | - | 从不写 | + +VARIANT 路径索引指用 `field_pattern` 声明的索引,以及每个被提取的子路径继承到的那份副本。一个段中每个路径各有一份这样的索引,因此在这里写 norms 的代价是 `行数 × 路径数` 字节。 + +显式设置该属性即可覆盖默认值: + +```sql +-- 为某个 VARIANT 路径保留记录长度归一化 +INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title", "norms" = "true") + +-- 关闭普通列上的记录长度归一化 +INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english", "norms" = "false") +``` + +索引没有 norms 时,打分会跳过记录长度归一化:公式中的 `b × |d| / avgdl` 项不再参与计算,得分只由词频和 IDF 决定。该属性对新写入的段生效;此前写入的段会保留原有的 norms,直到被 compaction 重写。 + ## 结果解读 理解打分结果有助于更准确地使用相关性排序: From 0d3f545c2f4f1401a8ec3a5c9acb645f81b9793b Mon Sep 17 00:00:00 2001 From: eldenmoon Date: Wed, 16 Sep 2026 15:31:53 +0800 Subject: [PATCH 2/5] [doc](inverted-index) Correct the norms default and document the BE config The norms index property defaults to true on every index. An index on a VARIANT path leaves norms out because the BE config inverted_index_skip_norms_for_variant (default true, changeable at runtime) skips them there, not because the property itself defaults to false on such an index. The property still decides on its own in either direction, and a subpath index inherits it from the index it comes from. Co-Authored-By: Claude Opus 5 --- .../sql-data-types/semi-structured/VARIANT.md | 6 +++--- .../table-design/index/inverted-index/scoring.md | 16 ++++++++-------- .../sql-data-types/semi-structured/VARIANT.md | 6 +++--- .../table-design/index/inverted-index/scoring.md | 16 ++++++++-------- 4 files changed, 22 insertions(+), 22 deletions(-) diff --git a/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md b/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md index 4e49970eb9c21..72bd428382ccf 100644 --- a/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md +++ b/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md @@ -462,9 +462,9 @@ SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris'; BM25 norms on VARIANT paths: -A tokenized index on a VARIANT path does not write BM25 norms by default, while a tokenized index on an ordinary column does. Norms store the record length used by BM25 and cost one byte per row for every indexed field, written even for rows that hold no value for that path. One segment keeps one index per path, so norms on a VARIANT column with many paths would cost `rows × paths` bytes. Queries are unaffected apart from ranking: record-length normalization is skipped, and `score()` depends on term frequency and IDF only. +A tokenized index writes BM25 norms by default, but the BE config `inverted_index_skip_norms_for_variant` (default `true`, changeable at runtime) leaves them out for an index on a VARIANT path. Norms store the record length used by BM25 and cost one byte per row for every indexed field, written even for rows that hold no value for that path. One segment keeps one index per path, so norms on a VARIANT column with many paths would cost `rows × paths` bytes. Queries are unaffected apart from ranking: record-length normalization is skipped, and `score()` depends on term frequency and IDF only. -Add `"norms" = "true"` to a path index when that path needs full BM25 scoring: +Add `"norms" = "true"` to a path index when that path needs full BM25 scoring. The property decides on its own whatever the config says, and an index without `field_pattern` passes it to every subpath it covers: ```sql CREATE TABLE IF NOT EXISTS tbl ( @@ -472,7 +472,7 @@ CREATE TABLE IF NOT EXISTS tbl ( v VARIANT<'title' : STRING, 'body_*' : STRING>, -- title is ranked with record-length normalization INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title", "norms" = "true"), - -- body_* keeps the default: no norms, no record-length normalization + -- body_* keeps the default: no norms while the config skips them, so no record-length normalization INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*") ); ``` diff --git a/docs/table-design/index/inverted-index/scoring.md b/docs/table-design/index/inverted-index/scoring.md index 2da24a9578bb8..6b7258c315358 100644 --- a/docs/table-design/index/inverted-index/scoring.md +++ b/docs/table-design/index/inverted-index/scoring.md @@ -115,17 +115,17 @@ When a query contains multiple terms, **the final score is the sum of the scores `|d|` comes from norms, which an index stores as one byte per row for every indexed field. The byte is written for every row of the segment, including rows that hold no value for that field, so an index that covers many sparse fields also pays for the rows it never matches. -Norms are controlled per index with the `norms` property: +Norms are controlled per index with the `norms` property, which defaults to `true`: -| Index | Default | Norms written | -| -------------------------------------- | ------- | ------------- | -| Tokenized index on an ordinary column | `true` | Yes | -| Tokenized index on a VARIANT path | `false` | No | -| Non-tokenized index | - | Never | +| Index | Norms written | +| -------------------------------------- | ------------------------------------------------------- | +| Tokenized index on an ordinary column | Yes | +| Tokenized index on a VARIANT path | No, while `inverted_index_skip_norms_for_variant` is on | +| Non-tokenized index | Never | -A VARIANT path index is an index declared with `field_pattern`, together with the copy of it that each extracted subpath inherits. One segment holds one such index per path, so writing norms there costs `rows × paths` bytes. +A VARIANT path index is an index declared with `field_pattern`, together with the copy of it that each extracted subpath inherits. One segment holds one such index per path, so writing norms there costs `rows × paths` bytes. The BE config `inverted_index_skip_norms_for_variant` (default `true`, changeable at runtime) therefore leaves norms out of those indexes; turn it off and a VARIANT path index behaves like any other index. -Set the property explicitly to override the default: +The `norms` property decides on its own either way, and the copy inherited by a subpath carries the property of the index it comes from: ```sql -- keep record-length normalization for one VARIANT path diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md index bfe3096f1e3c8..4d2e4992615df 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md @@ -462,9 +462,9 @@ SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris'; VARIANT 路径上的 BM25 norms: -VARIANT 路径上的分词索引默认不写 BM25 norms,而普通列上的分词索引会写。norms 保存 BM25 使用的记录长度,按行存储,每个被索引的字段每行 1 字节,即使该行在这个路径上没有值也会写入。一个段中每个路径各有一份索引,因此在路径数很多的 VARIANT 列上写 norms 的代价是 `行数 × 路径数` 字节。除排序外查询不受影响:仅跳过记录长度归一化,`score()` 只由词频和 IDF 决定。 +分词索引默认会写 BM25 norms,但 BE 配置项 `inverted_index_skip_norms_for_variant`(默认 `true`,可动态修改)会让 VARIANT 路径上的索引不写。norms 保存 BM25 使用的记录长度,按行存储,每个被索引的字段每行 1 字节,即使该行在这个路径上没有值也会写入。一个段中每个路径各有一份索引,因此在路径数很多的 VARIANT 列上写 norms 的代价是 `行数 × 路径数` 字节。除排序外查询不受影响:仅跳过记录长度归一化,`score()` 只由词频和 IDF 决定。 -如果某个路径需要完整的 BM25 打分,给该路径的索引加上 `"norms" = "true"`: +如果某个路径需要完整的 BM25 打分,给该路径的索引加上 `"norms" = "true"`。该属性的优先级高于配置项;不带 `field_pattern` 的索引会把它传给自己覆盖的每个子路径: ```sql CREATE TABLE IF NOT EXISTS tbl ( @@ -472,7 +472,7 @@ CREATE TABLE IF NOT EXISTS tbl ( v VARIANT<'title' : STRING, 'body_*' : STRING>, -- title 参与记录长度归一化 INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title", "norms" = "true"), - -- body_* 保持默认:不写 norms,不做记录长度归一化 + -- body_* 保持默认:配置项开启时不写 norms,也不做记录长度归一化 INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*") ); ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md index 67b1f19aae478..1f76032885461 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md @@ -115,17 +115,17 @@ avgdl = total_terms / total_rows 公式中的 `|d|` 来自 norms。索引会为每个被索引的字段按行存储 norms,每行 1 字节。这个字节对段内每一行都会写入,包括该字段没有值的行,因此覆盖大量稀疏字段的索引也要为从不命中的行付出空间。 -norms 由索引属性 `norms` 控制: +norms 由索引属性 `norms` 控制,该属性默认为 `true`: -| 索引 | 默认值 | 是否写 norms | -| ------------------------ | ------- | ------------ | -| 普通列上的分词索引 | `true` | 写 | -| VARIANT 路径上的分词索引 | `false` | 不写 | -| 非分词索引 | - | 从不写 | +| 索引 | 是否写 norms | +| ------------------------ | -------------------------------------------------------- | +| 普通列上的分词索引 | 写 | +| VARIANT 路径上的分词索引 | 不写(`inverted_index_skip_norms_for_variant` 开启时) | +| 非分词索引 | 从不写 | -VARIANT 路径索引指用 `field_pattern` 声明的索引,以及每个被提取的子路径继承到的那份副本。一个段中每个路径各有一份这样的索引,因此在这里写 norms 的代价是 `行数 × 路径数` 字节。 +VARIANT 路径索引指用 `field_pattern` 声明的索引,以及每个被提取的子路径继承到的那份副本。一个段中每个路径各有一份这样的索引,因此在这里写 norms 的代价是 `行数 × 路径数` 字节。BE 配置项 `inverted_index_skip_norms_for_variant`(默认 `true`,可动态修改)因此让这类索引不写 norms;关闭它之后,VARIANT 路径索引与普通索引行为一致。 -显式设置该属性即可覆盖默认值: +`norms` 属性在两种情况下都优先生效;子路径继承到的那份副本会带上它所来源的索引的属性: ```sql -- 为某个 VARIANT 路径保留记录长度归一化 From 8ffc6a9158cda988a8bfa9f12b972f976d725be1 Mon Sep 17 00:00:00 2001 From: eldenmoon Date: Wed, 16 Sep 2026 15:48:05 +0800 Subject: [PATCH 3/5] [doc](inverted-index) Follow the norms default and the config precedence An analyzed index writes BM25 norms on a VARIANT path just as it does on an ordinary column. The BE config inverted_index_skip_norms_for_variant is off by default and, once turned on, drops norms for every index on a VARIANT path whatever that index's norms property says. Both examples now set "norms" = "false", the only direction the property is still useful in. Co-Authored-By: Claude Opus 5 --- .../sql-data-types/semi-structured/VARIANT.md | 12 ++++++------ .../index/inverted-index/scoring.md | 18 +++++++++--------- .../sql-data-types/semi-structured/VARIANT.md | 12 ++++++------ .../index/inverted-index/scoring.md | 12 ++++++------ 4 files changed, 27 insertions(+), 27 deletions(-) diff --git a/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md b/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md index 72bd428382ccf..7906d2008fa56 100644 --- a/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md +++ b/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md @@ -462,18 +462,18 @@ SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris'; BM25 norms on VARIANT paths: -A tokenized index writes BM25 norms by default, but the BE config `inverted_index_skip_norms_for_variant` (default `true`, changeable at runtime) leaves them out for an index on a VARIANT path. Norms store the record length used by BM25 and cost one byte per row for every indexed field, written even for rows that hold no value for that path. One segment keeps one index per path, so norms on a VARIANT column with many paths would cost `rows × paths` bytes. Queries are unaffected apart from ranking: record-length normalization is skipped, and `score()` depends on term frequency and IDF only. +A tokenized index on a VARIANT path writes BM25 norms just like one on an ordinary column. Norms store the record length used by BM25 and cost one byte per row for every indexed field, written even for rows that hold no value for that path. One segment keeps one index per path, so norms on a VARIANT column with many paths cost `rows × paths` bytes. Turning on the BE config `inverted_index_skip_norms_for_variant` (default `false`, changeable at runtime) drops them for every index on a VARIANT path, whatever that index's `norms` property says. Where norms are missing, queries are unaffected apart from ranking: record-length normalization is skipped, and `score()` depends on term frequency and IDF only. -Add `"norms" = "true"` to a path index when that path needs full BM25 scoring. The property decides on its own whatever the config says, and an index without `field_pattern` passes it to every subpath it covers: +Add `"norms" = "false"` to a path index that does not need record-length ranking, and it stops paying for norms. An index without `field_pattern` passes the property to every subpath it covers: ```sql CREATE TABLE IF NOT EXISTS tbl ( k BIGINT, v VARIANT<'title' : STRING, 'body_*' : STRING>, - -- title is ranked with record-length normalization - INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title", "norms" = "true"), - -- body_* keeps the default: no norms while the config skips them, so no record-length normalization - INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*") + -- title keeps the default: norms, so score() applies record-length normalization + INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title"), + -- body_* gives up record-length normalization to save one byte per row per path + INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*", "norms" = "false") ); ``` diff --git a/docs/table-design/index/inverted-index/scoring.md b/docs/table-design/index/inverted-index/scoring.md index 6b7258c315358..3198caa8c914d 100644 --- a/docs/table-design/index/inverted-index/scoring.md +++ b/docs/table-design/index/inverted-index/scoring.md @@ -117,19 +117,19 @@ When a query contains multiple terms, **the final score is the sum of the scores Norms are controlled per index with the `norms` property, which defaults to `true`: -| Index | Norms written | -| -------------------------------------- | ------------------------------------------------------- | -| Tokenized index on an ordinary column | Yes | -| Tokenized index on a VARIANT path | No, while `inverted_index_skip_norms_for_variant` is on | -| Non-tokenized index | Never | +| Index | Norms written | +| -------------------------------------- | ---------------------------------------------------------------- | +| Tokenized index on an ordinary column | Yes, unless the index sets `"norms" = "false"` | +| Tokenized index on a VARIANT path | The same, unless the BE config below turns them off | +| Non-tokenized index | Never | -A VARIANT path index is an index declared with `field_pattern`, together with the copy of it that each extracted subpath inherits. One segment holds one such index per path, so writing norms there costs `rows × paths` bytes. The BE config `inverted_index_skip_norms_for_variant` (default `true`, changeable at runtime) therefore leaves norms out of those indexes; turn it off and a VARIANT path index behaves like any other index. +A VARIANT path index is an index declared with `field_pattern`, together with the copy of it that each extracted subpath inherits. One segment holds one such index per path, so writing norms there costs `rows × paths` bytes. Turning on the BE config `inverted_index_skip_norms_for_variant` (default `false`, changeable at runtime) drops norms for every index on a VARIANT path, whatever that index's `norms` property says, so a cluster can reclaim that space without rewriting its index definitions. -The `norms` property decides on its own either way, and the copy inherited by a subpath carries the property of the index it comes from: +The `norms` property decides per index, and the copy inherited by a subpath carries the property of the index it comes from: ```sql --- keep record-length normalization for one VARIANT path -INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title", "norms" = "true") +-- drop record-length normalization for one VARIANT path +INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*", "norms" = "false") -- drop it for an ordinary column INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english", "norms" = "false") diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md index 4d2e4992615df..999f57b14b0f3 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md @@ -462,18 +462,18 @@ SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris'; VARIANT 路径上的 BM25 norms: -分词索引默认会写 BM25 norms,但 BE 配置项 `inverted_index_skip_norms_for_variant`(默认 `true`,可动态修改)会让 VARIANT 路径上的索引不写。norms 保存 BM25 使用的记录长度,按行存储,每个被索引的字段每行 1 字节,即使该行在这个路径上没有值也会写入。一个段中每个路径各有一份索引,因此在路径数很多的 VARIANT 列上写 norms 的代价是 `行数 × 路径数` 字节。除排序外查询不受影响:仅跳过记录长度归一化,`score()` 只由词频和 IDF 决定。 +VARIANT 路径上的分词索引和普通列上的一样会写 BM25 norms。norms 保存 BM25 使用的记录长度,按行存储,每个被索引的字段每行 1 字节,即使该行在这个路径上没有值也会写入。一个段中每个路径各有一份索引,因此在路径数很多的 VARIANT 列上写 norms 的代价是 `行数 × 路径数` 字节。打开 BE 配置项 `inverted_index_skip_norms_for_variant`(默认 `false`,可动态修改)之后,VARIANT 路径上的所有索引都不再写 norms,无论该索引的 `norms` 属性如何。没有 norms 时,除排序外查询不受影响:仅跳过记录长度归一化,`score()` 只由词频和 IDF 决定。 -如果某个路径需要完整的 BM25 打分,给该路径的索引加上 `"norms" = "true"`。该属性的优先级高于配置项;不带 `field_pattern` 的索引会把它传给自己覆盖的每个子路径: +如果某个路径不需要按记录长度排序,给它的索引加上 `"norms" = "false"` 即可省下这部分空间。不带 `field_pattern` 的索引会把该属性传给自己覆盖的每个子路径: ```sql CREATE TABLE IF NOT EXISTS tbl ( k BIGINT, v VARIANT<'title' : STRING, 'body_*' : STRING>, - -- title 参与记录长度归一化 - INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title", "norms" = "true"), - -- body_* 保持默认:配置项开启时不写 norms,也不做记录长度归一化 - INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*") + -- title 保持默认:写 norms,score() 参与记录长度归一化 + INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title"), + -- body_* 放弃记录长度归一化,省下每行每路径 1 字节 + INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*", "norms" = "false") ); ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md index 1f76032885461..bd6d45ce70dfa 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md @@ -119,17 +119,17 @@ norms 由索引属性 `norms` 控制,该属性默认为 `true`: | 索引 | 是否写 norms | | ------------------------ | -------------------------------------------------------- | -| 普通列上的分词索引 | 写 | -| VARIANT 路径上的分词索引 | 不写(`inverted_index_skip_norms_for_variant` 开启时) | +| 普通列上的分词索引 | 写,除非索引设置了 `"norms" = "false"` | +| VARIANT 路径上的分词索引 | 同上,除非下面的 BE 配置项把它们关掉 | | 非分词索引 | 从不写 | -VARIANT 路径索引指用 `field_pattern` 声明的索引,以及每个被提取的子路径继承到的那份副本。一个段中每个路径各有一份这样的索引,因此在这里写 norms 的代价是 `行数 × 路径数` 字节。BE 配置项 `inverted_index_skip_norms_for_variant`(默认 `true`,可动态修改)因此让这类索引不写 norms;关闭它之后,VARIANT 路径索引与普通索引行为一致。 +VARIANT 路径索引指用 `field_pattern` 声明的索引,以及每个被提取的子路径继承到的那份副本。一个段中每个路径各有一份这样的索引,因此在这里写 norms 的代价是 `行数 × 路径数` 字节。打开 BE 配置项 `inverted_index_skip_norms_for_variant`(默认 `false`,可动态修改)之后,VARIANT 路径上的所有索引都不再写 norms,无论该索引的 `norms` 属性如何,这样集群不必改写索引定义就能收回这部分空间。 -`norms` 属性在两种情况下都优先生效;子路径继承到的那份副本会带上它所来源的索引的属性: +`norms` 属性按索引生效;子路径继承到的那份副本会带上它所来源的索引的属性: ```sql --- 为某个 VARIANT 路径保留记录长度归一化 -INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title", "norms" = "true") +-- 关闭某个 VARIANT 路径上的记录长度归一化 +INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*", "norms" = "false") -- 关闭普通列上的记录长度归一化 INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english", "norms" = "false") From de97482a2581e852cf8ef8d89393914554163451 Mon Sep 17 00:00:00 2001 From: eldenmoon Date: Wed, 16 Sep 2026 18:10:14 +0800 Subject: [PATCH 4/5] [doc](inverted-index) Note the upgrade order for norms and how rows without norms score A row without norms is scored as if its length were avgdl, so the whole length factor 1 - b + b * |d| / avgdl becomes 1, rather than one term of it dropping out. Turning inverted_index_skip_norms_for_variant on, or setting "norms" = "false", should wait until every BE is upgraded, because an older BE cannot score an index written without norms. Co-Authored-By: Claude Opus 5 --- docs/table-design/index/inverted-index/scoring.md | 2 +- .../current/table-design/index/inverted-index/scoring.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/table-design/index/inverted-index/scoring.md b/docs/table-design/index/inverted-index/scoring.md index 3198caa8c914d..a1ae28ffc4444 100644 --- a/docs/table-design/index/inverted-index/scoring.md +++ b/docs/table-design/index/inverted-index/scoring.md @@ -135,7 +135,7 @@ INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english", "norms" = "false") ``` -When an index has no norms, scoring skips record-length normalization: the `b × |d| / avgdl` term drops out and the score depends on term frequency and IDF only. The property applies to newly written segments; segments written earlier keep their norms until compaction rewrites them. +When an index has no norms, scoring skips record-length normalization: every row is scored as if its length were `avgdl`, so `1 - b + b × |d| / avgdl` becomes 1 and the score depends on term frequency and IDF only. The property and the config apply to newly written segments; segments written earlier keep their norms until compaction rewrites them. Turn the config on, or set `"norms" = "false"`, only after every BE has been upgraded to a version that supports them, because an older BE cannot score an index written without norms. ## Interpreting the Results diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md index bd6d45ce70dfa..bf9af15533a02 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md @@ -135,7 +135,7 @@ INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english", "norms" = "false") ``` -索引没有 norms 时,打分会跳过记录长度归一化:公式中的 `b × |d| / avgdl` 项不再参与计算,得分只由词频和 IDF 决定。该属性对新写入的段生效;此前写入的段会保留原有的 norms,直到被 compaction 重写。 +索引没有 norms 时,打分会跳过记录长度归一化:每行都按平均长度 `avgdl` 计算,`1 - b + b × |d| / avgdl` 恒为 1,得分只由词频和 IDF 决定。该属性和配置项只对新写入的段生效;此前写入的段会保留原有的 norms,直到被 compaction 重写。请在所有 BE 都升级到支持它们的版本之后,再打开配置项或设置 `"norms" = "false"`,因为旧版本的 BE 无法对没有 norms 的索引打分。 ## 结果解读 From 5e4c4627d9fdb770a2961eff60a952a1a6d411df Mon Sep 17 00:00:00 2001 From: eldenmoon Date: Wed, 16 Sep 2026 21:28:14 +0800 Subject: [PATCH 5/5] [doc](inverted-index) Say that score() needs norms from every segment A query that computes score() on a tokenized index now fails when any segment it reads has no norms for the field, including a table where only some segments were written without them, while MATCH filtering is unaffected. Describe that, and present "norms" = "false" and inverted_index_skip_norms_for_variant as options for indexes that are never ranked with score(). Co-Authored-By: Claude Opus 5 --- .../sql-data-types/semi-structured/VARIANT.md | 8 ++++---- docs/table-design/index/inverted-index/scoring.md | 2 +- .../sql-data-types/semi-structured/VARIANT.md | 8 ++++---- .../current/table-design/index/inverted-index/scoring.md | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md b/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md index 7906d2008fa56..b64968175700f 100644 --- a/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md +++ b/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md @@ -462,17 +462,17 @@ SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris'; BM25 norms on VARIANT paths: -A tokenized index on a VARIANT path writes BM25 norms just like one on an ordinary column. Norms store the record length used by BM25 and cost one byte per row for every indexed field, written even for rows that hold no value for that path. One segment keeps one index per path, so norms on a VARIANT column with many paths cost `rows × paths` bytes. Turning on the BE config `inverted_index_skip_norms_for_variant` (default `false`, changeable at runtime) drops them for every index on a VARIANT path, whatever that index's `norms` property says. Where norms are missing, queries are unaffected apart from ranking: record-length normalization is skipped, and `score()` depends on term frequency and IDF only. +A tokenized index on a VARIANT path writes BM25 norms just like one on an ordinary column. Norms store the record length used by BM25 and cost one byte per row for every indexed field, written even for rows that hold no value for that path. One segment keeps one index per path, so norms on a VARIANT column with many paths cost `rows × paths` bytes. Turning on the BE config `inverted_index_skip_norms_for_variant` (default `false`, changeable at runtime) drops them for every index on a VARIANT path, whatever that index's `norms` property says. Without norms, `MATCH` filtering still works, but a query that computes `score()` on that path returns an error, also while only some of the table's segments lack norms. -Add `"norms" = "false"` to a path index that does not need record-length ranking, and it stops paying for norms. An index without `field_pattern` passes the property to every subpath it covers: +Add `"norms" = "false"` to a path index that is never ranked with `score()`, and it stops paying for norms. An index without `field_pattern` passes the property to every subpath it covers: ```sql CREATE TABLE IF NOT EXISTS tbl ( k BIGINT, v VARIANT<'title' : STRING, 'body_*' : STRING>, - -- title keeps the default: norms, so score() applies record-length normalization + -- title keeps the default: norms, so it can be ranked with score() INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title"), - -- body_* gives up record-length normalization to save one byte per row per path + -- body_* is only filtered, never ranked: no norms, one byte less per row per path INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*", "norms" = "false") ); ``` diff --git a/docs/table-design/index/inverted-index/scoring.md b/docs/table-design/index/inverted-index/scoring.md index a1ae28ffc4444..093b3c4fbbfd9 100644 --- a/docs/table-design/index/inverted-index/scoring.md +++ b/docs/table-design/index/inverted-index/scoring.md @@ -135,7 +135,7 @@ INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english", "norms" = "false") ``` -When an index has no norms, scoring skips record-length normalization: every row is scored as if its length were `avgdl`, so `1 - b + b × |d| / avgdl` becomes 1 and the score depends on term frequency and IDF only. The property and the config apply to newly written segments; segments written earlier keep their norms until compaction rewrites them. Turn the config on, or set `"norms" = "false"`, only after every BE has been upgraded to a version that supports them, because an older BE cannot score an index written without norms. +Relevance scoring needs norms: a query that computes `score()` on a tokenized index returns an error when any segment it reads has no norms for that field, which includes a table where only some segments were written without them. `MATCH_*` filtering is not affected. Set `"norms" = "false"`, or turn the config on, only for indexes that are never ranked with `score()`. The property and the config apply to newly written segments; segments written earlier keep their norms until compaction rewrites them. Change either one only after every BE has been upgraded to a version that supports them. ## Interpreting the Results diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md index 999f57b14b0f3..753dae5a98402 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md @@ -462,17 +462,17 @@ SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris'; VARIANT 路径上的 BM25 norms: -VARIANT 路径上的分词索引和普通列上的一样会写 BM25 norms。norms 保存 BM25 使用的记录长度,按行存储,每个被索引的字段每行 1 字节,即使该行在这个路径上没有值也会写入。一个段中每个路径各有一份索引,因此在路径数很多的 VARIANT 列上写 norms 的代价是 `行数 × 路径数` 字节。打开 BE 配置项 `inverted_index_skip_norms_for_variant`(默认 `false`,可动态修改)之后,VARIANT 路径上的所有索引都不再写 norms,无论该索引的 `norms` 属性如何。没有 norms 时,除排序外查询不受影响:仅跳过记录长度归一化,`score()` 只由词频和 IDF 决定。 +VARIANT 路径上的分词索引和普通列上的一样会写 BM25 norms。norms 保存 BM25 使用的记录长度,按行存储,每个被索引的字段每行 1 字节,即使该行在这个路径上没有值也会写入。一个段中每个路径各有一份索引,因此在路径数很多的 VARIANT 列上写 norms 的代价是 `行数 × 路径数` 字节。打开 BE 配置项 `inverted_index_skip_norms_for_variant`(默认 `false`,可动态修改)之后,VARIANT 路径上的所有索引都不再写 norms,无论该索引的 `norms` 属性如何。没有 norms 时 `MATCH` 过滤照常可用,但在该路径上计算 `score()` 的查询会报错;表中只有部分段缺少 norms 时也一样。 -如果某个路径不需要按记录长度排序,给它的索引加上 `"norms" = "false"` 即可省下这部分空间。不带 `field_pattern` 的索引会把该属性传给自己覆盖的每个子路径: +如果某个路径从不用 `score()` 排序,给它的索引加上 `"norms" = "false"` 即可省下这部分空间。不带 `field_pattern` 的索引会把该属性传给自己覆盖的每个子路径: ```sql CREATE TABLE IF NOT EXISTS tbl ( k BIGINT, v VARIANT<'title' : STRING, 'body_*' : STRING>, - -- title 保持默认:写 norms,score() 参与记录长度归一化 + -- title 保持默认:写 norms,可以用 score() 排序 INDEX idx_title(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "title"), - -- body_* 放弃记录长度归一化,省下每行每路径 1 字节 + -- body_* 只做过滤、不排序:不写 norms,每行每路径省 1 字节 INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern" = "body_*", "norms" = "false") ); ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md index bf9af15533a02..9a73c4ebfe0b9 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md @@ -135,7 +135,7 @@ INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english", "field_pattern INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english", "norms" = "false") ``` -索引没有 norms 时,打分会跳过记录长度归一化:每行都按平均长度 `avgdl` 计算,`1 - b + b × |d| / avgdl` 恒为 1,得分只由词频和 IDF 决定。该属性和配置项只对新写入的段生效;此前写入的段会保留原有的 norms,直到被 compaction 重写。请在所有 BE 都升级到支持它们的版本之后,再打开配置项或设置 `"norms" = "false"`,因为旧版本的 BE 无法对没有 norms 的索引打分。 +相关性打分依赖 norms:在分词索引上计算 `score()` 的查询,只要读到的任意一个段缺少该字段的 norms 就会报错,只有部分段缺少 norms 的表也一样。`MATCH_*` 过滤不受影响。请只对从不用 `score()` 排序的索引设置 `"norms" = "false"` 或打开该配置项。该属性和配置项只对新写入的段生效;此前写入的段会保留原有的 norms,直到被 compaction 重写。请在所有 BE 都升级到支持它们的版本之后再修改。 ## 结果解读