MDEV-40168 wip by mariadb-YuchenPei · Pull Request #5620 · MariaDB/server

mariadb-YuchenPei · 2026-09-01T06:31:31Z

TODOs on top of those in the patch diff:

EXPLAIN output should not say fulltext
check type match to avoid false negative / positive bugs in mysql
transcode the value into the index charset in mvi_encode_key

CLAassistant · 2026-09-01T06:31:40Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ spetrunia
❌ mariadb-YuchenPei
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

spetrunia · 2026-09-01T09:05:42Z

+t1	CREATE TABLE `t1` (
+  `c` int(11) DEFAULT NULL,
+  `j` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid(`j`))
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_uca1400_ai_ci


So, the index is now shown in SHOW CREATE TABLE ?

rather is NOT shown in SHOW CREATE TABLE

spetrunia · 2026-09-03T13:19:24Z

For the record: index is not visible in information_schema:

create table t25 (
  js json,
  key idx ((CAST(json_extract(js, '$.tags') AS CHAR(6) ARRAY)))
)engine=innodb;
select * from information_schema.statistics where table_name='t25'

gives nothing.

spetrunia · 2026-09-03T13:25:47Z

And this crashes:

create table t25 (
  js json,
  key idx ((CAST(json_extract(js, '$.tags') AS CHAR(6) ARRAY)))
)engine=innodb;
insert into t25 values ('{}');

mariadb-YuchenPei · 2026-09-04T00:49:48Z

For the record: index is not visible in information_schema:

create table t25 (
  js json,
  key idx ((CAST(json_extract(js, '$.tags') AS CHAR(6) ARRAY)))
)engine=innodb;
select * from information_schema.statistics where table_name='t25'

gives nothing.

Isn't this the same problem as your other comment #5620 (comment)?

TODOs on top of those in the patch diff: - EXPLAIN output should not say fulltext - check type match to avoid false negative / positive bugs in mysql - transcode the value into the index charset in mvi_encode_key

mariadb-YuchenPei · 2026-09-04T01:49:48Z

And this crashes:

create table t25 (
  js json,
  key idx ((CAST(json_extract(js, '$.tags') AS CHAR(6) ARRAY)))
)engine=innodb;
insert into t25 values ('{}');

Fixed now

Item_func_json_contains::mvi_analyze() and ::create_ft_for_mvi() were near-identical: both checked the arguments, looked up the matching MVI, parsed the constant second argument and ran the same scan loop calling encode_mvi_key(). They differed only in what they did with each encoded key. Move all of that into get_mvi_access(), which returns an Mvi_access, and give Mvi_access two methods: - add_key(), to collect one encoded element key, - create_ft_item(), to build the MATCH vcol AGAINST ('+encoded_foo +encoded_bar ...' IN BOOLEAN MODE) item. It honors Mvi_access::conjunctive, so JSON_OVERLAPS will get the OR form for free. mvi_analyze() and create_ft_for_mvi() are now thin wrappers around get_mvi_access(). This also fixes a memory leak: the encoded keys were copied with String::copy(), giving each String in Mvi_access::encoded a heap buffer that is never freed (the Strings live on the MEM_ROOT, so their destructors never run). Copy the keys onto the MEM_ROOT instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

JSON_CONTAINS() over a multi-valued index used to be optimized by rewriting the WHERE clause: setup_mvi_for_join() injected a synthetic MATCH vcol AGAINST ('+k1 +k2' IN BOOLEAN MODE) into join->conds and into select_lex->ftfunc_list, and the normal fulltext machinery then picked it up as JT_FT access. The injected item showed up in the plan and in the condition even though the user never wrote it, and because it became ordinary ref access the scan was never costed against the alternatives - it won by being in the WHERE clause. Introduce QUICK_MVI_SELECT (QS_TYPE_MVI), a QUICK_SELECT_I that drives the fulltext index directly through the handler API. Unlike FT_SELECT there is no Item_func_match to have created the FT_INFO, so the quick select creates it in reset() with ft_init_ext() and frees it with close_search() in its destructor. Mvi_access::create_ft_item() is replaced by build_ft_query(), which builds just the query string. The analysis in setup_mvi_quick() is now kept: Mvi_context moves to the header, is allocated on the mem_root and stored as JOIN::mvi_ctx, where JOIN::get_mvi_access_for_table() looks it up. get_quick_record_count() builds the quick select before test_quick_select() and keeps whichever of the two is cheaper; test_quick_select() itself is untouched, so the MVI quick is held in a local across the call (it deletes select->quick on entry). The same save/compare is done around the second test_quick_select() call in make_join_select(), which a LIMIT can reach. A fulltext key never gets a bit in const_keys or keys, so mark the MVI key of every table that has an access: the const_keys bit is what lets the range analysis run for that table at all, the keys bit puts the index into EXPLAIN's possible_keys. Collect the accesses from the top-level AND-parts of the WHERE clause only, instead of walking the whole condition. An MVI scan reads just the rows the index matches, so it is only valid for a predicate that must hold for every row of the result: for json_contains(j1->'$.tags','"a"') OR json_contains(j2->'$.tags','"a"') scanning either index would drop the rows that only match the other branch. The deleted add_ft_for_mvi() refused COND_OR_FUNC for the same reason; walking the condition tree lost that, which only became visible once the accesses were actually used. Costs are placeholders (records=10, read_time=0.001) until the engine can estimate a fulltext search. Note that while there is no estimate, an MVI access is also taken when test_quick_select() produced no quick select at all, without comparing it to the cost of a table scan. TODO: This doesn't handle UPDATE/DELETE! Should it be put into check_quick() call? Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…elects.

get_best_mvi_access() picked an Mvi_access and wrapped it in a QUICK_MVI_SELECT without recording anything, so there was no way to see which index was chosen or what it would search that index for: the rows_estimation trace showed a range_analysis that found nothing, and then a plan using a key the trace never mentioned. Print a "multi_value_index_use" object: { "table": "t1", "index": "idx", "ranges": ["616161"] } Mvi_access::print_json() fills in the index and the element keys, following TRP_RANGE::trace_basic_info(): same "index" / "ranges" member names, so an MVI entry reads like a range scan's. The keys are printed in their encoded form, which is not readable. That is what is stored in the index and what we search for, so it is still the useful thing to print; making it readable can come later. It is plain ASCII (hex plus the xx/xxxx padding from encode_mvi_key()), so it needs no JSON escaping. get_best_mvi_access() runs inside the "rows_estimation" array, so the named object needs an object of its own around it, the same way make_join_statistics() and the sel_arg_alloc_limit_hit trace do it. Without it the writer hits an assertion in Single_line_formatting_helper::on_add_member(). The new test is a separate file because optimizer trace tests need not_embedded.inc, and putting that in multi_valued_index.test would skip the whole feature test on embedded builds. It cross-checks the printed keys against mvi_encode() over the indexed column, which produces the tokens the index is actually built from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

It collects Mv_index objects, not vcols. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

JOIN_TAB held the whole Mvi_context the analysis produced, but outside setup_mvi_access_for_table() the only thing ever read out of it was mvi_ctx->best. The rest is scratch: indexes feeds get_mvi_index(), accesses is what the last-wins loop picks best out of, and thd is there for the mvi_analyze() callbacks. So JOIN_TAB keeps the access itself, and the context becomes a local of setup_mvi_access_for_table() - which is what the TODO there asked for: nothing is allocated for a table that has no MVI key, or whose condition yields no access. The access outliving the context is safe because neither it nor the Mv_index it refers to belongs to the context: both are allocated on the MEM_ROOT, and the lists only hold link nodes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

create table t1 (c int, j json, key idx ((CAST(j->'$.tags' AS CHAR(6) ARRAY)))); printed the two columns and no index at all. Nobody had decided to hide it: it inherited invisibility from the internal column that backs it. The grammar makes DB_MVI_<n> INVISIBLE_FULL, init_from_binary_frm_image() turns a hidden key part into a hidden key, and store_create_info() skips keys with HA_INVISIBLE_KEY. Long unique hash keys - the other kind of key built over a column the user cannot name - are already exempted from that; exempt the multi-valued index the same way, and print it as KEY `idx` ((cast(json_extract(`j`,'$.tags') as char(6) array))) which is the expression it was declared with and all it takes to re-create it. The internal column stays out of the output: it cannot be printed as a column, since there is no syntax that would recreate the pairing. Item_func_mvi_encode::print() cannot produce that form. Its output is what pack_expression() puts in the FRM, and that is read back as a call of mvi_encode(), the only form the parser accepts outside an index definition. So the CAST spelling gets a printer of its own, sharing the type printing. SHOW INDEX and I_S.STATISTICS list the key now too - the key part is let through the invisibility filter - so they no longer need debug_dbug=test_invisible_index, and the tests stop setting it (it also injected a stray invisible1 column and key into their output). The same HA_INVISIBLE_KEY drove mysql_prepare_alter_table(), which drops such keys from the list of keys carried into the rebuilt table - and INVISIBLE_FULL columns from the list of columns. So ALTER TABLE t1 ADD COLUMN x INT; silently dropped the index. The key survives now, and its column is carried over with it, for exactly as long as the key lives: DROP KEY takes the column with it, so the name is free again afterwards. While at it, make_internal_field_name() looped forever when create_list is empty: dup_found started at true and the loop that clears it does not run. The MVI path is the only caller that can hit that, and it does - with ALTER TABLE ... ADD KEY ((CAST(... ARRAY))), which used to hang the server and now works. A fulltext key over several arrays, KEY idx ((CAST(j->'$.a' AS CHAR(6) ARRAY)), (CAST(j->'$.b' AS CHAR(6) ARRAY))) has no single expression to print and no syntax of its own to be read back, so it stays hidden, exactly as before. The optimizer still uses its parts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

create table t1 (j json, key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY)), (CAST(j->'$.b' AS CHAR(6) ARRAY)))); was accepted without a word. Each ARRAY key part gets an internal column of its own, and they all became key parts of one fulltext key: the tokens of both arrays end up mixed in a single index, and the optimizer would then search that index for the keys of one array and get the rows of the other as well. There is also no way to show such a key, or to read one back. init_key_part_spec() now rejects a key that has an ARRAY key part and more than one key part, on both the CREATE TABLE and the ALTER TABLE path. The other order, KEY idx (c,(CAST(... ARRAY))), was already rejected: the ARRAY part makes the key FULLTEXT, and `c' cannot be part of one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

create table t1 (j json, unique key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY)))); create table t1 (j json, primary key ((CAST(j->'$.a' AS CHAR(6) ARRAY)))); create table t1 (j json, fulltext key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY)))); were all accepted without a word, and all produced the same thing: a plain index. The key type the user wrote was simply overwritten with Key::FULLTEXT, so the table ended up with no unique constraint, or no primary key, or with a FULLTEXT index that MATCH() finds nothing in - the index holds encoded element keys, not the text. What the server builds for an ARRAY is a fulltext index over those encoded elements, and it can only mean what a plain KEY means. Say so: reject any other type, in the grammar, before that overwrite loses what was asked for. CONSTRAINT ... UNIQUE and the ALTER TABLE forms go the same way. SPATIAL and VECTOR are already syntax errors for an ARRAY key part; they are in the switch anyway so it stays exhaustive. FOREIGN KEY is unaffected: it builds its key with Key::MULTIPLE and cannot be told apart here. It is rejected, further down, by the engine - "Foreign key constraint is incorrectly formed". The count of key parts is now also checked in the grammar, and not only in init_key_part_spec(). Otherwise the first ARRAY part of a two-part key sets the type to FULLTEXT, and the second part reports "Incorrect usage of FULLTEXT and ARRAY" for a key nobody declared FULLTEXT. As a side effect KEY idx (c,(CAST(... ARRAY))) now gives the same "max 1 parts" error as the other orders, instead of ER_BAD_FT_COLUMN for `c'. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

mysql_prepare_alter_table() computed it at the top of the key loop and read it ~290 lines below, at the one place that wants it. Nothing in between can change the answer, and the early call also ran for keys that never reach the assignment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

multi_valued_key_part: built a whole schema object in its action: the hidden DB_MVI_<n> column with its MVI_ENCODE() vcol, the rewrite of the key into an invisible fulltext one, and the key part naming the column. That is DDL, and it belongs with the rest of the multi-valued index code, not in sql_yacc.yy where nobody reading opt_multi_valued_index.cc will find it. It becomes add_mvi_key_part(), which returns the key part or NULL if it raised an error, and the production is three lines. check_mvi_key_type() follows its only caller and turns static, so sql_table.h loses a declaration and the forward `class Key;` that existed only for it. No behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… a table based on costs Deduplicate while we are at it. Previously we simply chose the last access. Now we merge conjunctives and keep all mvi accesses on the same table, and choose the best one at costing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

An element that is itself an array is flattened, the same way MVI_ENCODE flattens the document. JSON_OVERLAPS does not flatten: it only matches such an element against a document element that is an array too, compared whole (json_compare_arrays_in_order()). The flattening here is still safe as it will produce only false positives that will be eliminated by a recheck. The only exception is when the nested array yields no key at all i.e. [], [[]], [[],[]], [[[]]], etc. Such an element may match a document element that has no key of ours either, so nothing we could search for would find that row. Give up in this case, as for a failed encoding. select json_overlaps('[[]]', '[[], "aaa"]'); is true, but the document has no key in the index and the scan for the "aaa" key does not return it. Give up on the access in that case, as we already do for an element that cannot be encoded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Mvi_array_iterator::start() and next() return the next thing the walk found -- a key, an element with no key, a nested array opened or closed, or one of the ways the walk ends -- and the caller loops over them with its own control flow: MVI_ENCODE goes back to its gotos, collect_mvi_keys() to plain returns, and the state it kept in a visitor is local variables again. The buffer to encode into is a constructor argument, so MVI_ENCODE still gets its keys written straight into the document it is building. No functional changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A key image that does not fit in a fulltext token was rejected, which costs the index for that value entirely: no key in the document, and an OVERLAPS that mentions the value gives up on the index altogether. Cut the image down to MVI_KEY_IMAGE_MAX_LEN instead. Two values that agree on that many bytes then share a key, which costs false positives and nothing else, since the predicate is rechecked on every row the index produces. This is what the non-binary path has always done -- strnxfrm() is asked for exactly that many bytes of weights and cannot return more -- so it makes the binary path, which is the one a JSON column takes, behave the same. No wildcard is needed in the fulltext query for this. Both sides of the index cut at the same point, so the key a search builds for a long value is the same string as the token the document has for it, and an exact term match finds it. A trailing '*' would only widen the term to keys that are longer than the one searched for, and after the cut there are none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch 2 times, most recently from 8a9c029 to 21cf57e Compare September 1, 2026 06:51

gkodinov added the MariaDB Corporation label Sep 1, 2026

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch from 21cf57e to d3452c7 Compare September 1, 2026 07:51

spetrunia self-requested a review September 1, 2026 09:05

spetrunia reviewed Sep 1, 2026

View reviewed changes

Comment thread mysql-test/main/multi_valued_index.result

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch from d3452c7 to 7cfaf2b Compare September 1, 2026 10:09

spetrunia reviewed Sep 2, 2026

View reviewed changes

Comment thread sql/opt_multi_valued_index.cc

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch 4 times, most recently from 344ad21 to 08d0e46 Compare September 3, 2026 07:45

spetrunia reviewed Sep 3, 2026

View reviewed changes

Comment thread sql/sql_yacc.yy Outdated

MDEV-40168 [wip] Add multi valued index over fulltext

78f3529

TODOs on top of those in the patch diff: - EXPLAIN output should not say fulltext - check type match to avoid false negative / positive bugs in mysql - transcode the value into the index charset in mvi_encode_key

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch from 6ff4ac9 to 78f3529 Compare September 4, 2026 01:49

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch from e7ff23f to d14f631 Compare September 4, 2026 06:35

MDEV-40168 [wip] mvi quick

2b86847

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch from d14f631 to 2b86847 Compare September 4, 2026 07:06

spetrunia and others added 6 commits September 4, 2026 14:06

Add comments

99b2b44

Factor out common code into get_mvi_index()

f589644

Make optimizer trace print "range", not "index_merge" for MVI quick s…

39ceedf

…elects.

spetrunia and others added 5 commits September 7, 2026 13:21

Rename collect_mvi_vcols_for_table to collect_mvi_indexes_for_table

aef91d6

It collects Mv_index objects, not vcols. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch 2 times, most recently from 61c85df to 8a77a3a Compare September 9, 2026 06:24

mariadb-YuchenPei and others added 9 commits September 10, 2026 10:54

MDEV-40168 Fix cast to int arrays

bdf696e

MDEV-40168 nested array handling and json validation

5bf3ce5

Undo whitespace changes to reduce diff size

a2d80ea

opt_mvi_jsonfuncs.cc: Move the code, const-ify, add comments.

2d4e9d5

Improve comments, formatting.

b8ccf8b

MDEV-40168 Add testcases for when there's both a ft index and an mvi

c5bbce6

MDEV-40168 Add some DDL tests

f490dd7

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch 7 times, most recently from 77c6422 to 7e32105 Compare September 11, 2026 02:02

mariadb-YuchenPei and others added 2 commits September 11, 2026 13:52

More comments, code readability. No functional changes.

b84a36f

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch from 7e32105 to db8b2a9 Compare September 11, 2026 03:55

mariadb-YuchenPei and others added 3 commits September 11, 2026 13:58

mariadb-YuchenPei force-pushed the bb-main-mdev-40168 branch from db8b2a9 to 3f4d449 Compare September 11, 2026 03:59

Navigation Menu

Uh oh!

Uh oh!

MDEV-40168 wip - #5620

MDEV-40168 wip#5620
mariadb-YuchenPei wants to merge 39 commits into
mainfrom
bb-main-mdev-40168

mariadb-YuchenPei commented Sep 1, 2026

Uh oh!

CLAassistant commented Sep 1, 2026 •

edited

Loading

Uh oh!

spetrunia Sep 1, 2026

Uh oh!

spetrunia Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spetrunia commented Sep 3, 2026

Uh oh!

spetrunia commented Sep 3, 2026

Uh oh!

mariadb-YuchenPei commented Sep 4, 2026

Uh oh!

mariadb-YuchenPei commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

mariadb-YuchenPei commented Sep 1, 2026

Uh oh!

CLAassistant commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spetrunia Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

spetrunia Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spetrunia commented Sep 3, 2026

Uh oh!

spetrunia commented Sep 3, 2026

Uh oh!

mariadb-YuchenPei commented Sep 4, 2026

Uh oh!

mariadb-YuchenPei commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

CLAassistant commented Sep 1, 2026 •

edited

Loading