MDEV-39918 Fix crash in degenerate jtbm semi-join with recursive CTE/…

[akshatnehra]

Description

This PR fixes MDEV-39918 where MariaDB crashes with an assertion failure (or segfault in non-debug builds) when a recursive CTE or UNION subquery is used in an IN predicate that triggers the degenerate JTBM semi-join path.

execute_degenerate_jtbm_semi_join() assumes the subquery engine is SINGLE_SELECT_ENGINE and casts unconditionally. However, recursive CTEs, UNIONs, EXCEPTs, and INTERSECTs use UNION_ENGINE even when the first SELECT has no tables (table_count == 0). The existing condition only checked table_count, not the engine type.

The fix involves:

1. Adding an engine type check (SINGLE_SELECT_ENGINE) at both call sites before entering the degenerate path. Non-matching subqueries fall through to normal JTBM materialization.

How can this PR be tested?

Run the MTR test:

build/mysql-test/mtr main.mdev_39918

Or manually (this will crash without the fix):

SELECT 1 FROM (SELECT 1 AS x) AS t
WHERE x IN (
  WITH RECURSIVE x(x) AS (SELECT 10 AS x)
  SELECT ROW_NUMBER() OVER (ORDER BY AVG(x))
  FROM (SELECT 1 AS x WHERE 1=0) AS d WHERE x
);

Results from my testing

1. Before changes (server crashes)

mariadbd: /workspace/mariadb-server/sql/opt_subselect.cc:6455:
  bool execute_degenerate_jtbm_semi_join(THD *, TABLE_LIST *,
  Item_in_subselect *, List<Item> &):
  Assertion `subq_pred->engine->engine_type() ==
  subselect_engine::SINGLE_SELECT_ENGINE' failed.

260612 22:23:55 [ERROR] ./sql/mariadbd got signal 6 ;
Sorry, we probably made a mistake, and this is a bug.

Client gets: ERROR 2026 (HY000): TLS/SSL error: unexpected eof while reading (connection lost due to server crash).

2. After changes

MariaDB> SELECT 1 FROM (SELECT 1 AS x) AS t
    -> WHERE x IN (
    ->   WITH RECURSIVE x(x) AS (SELECT 10 AS x)
    ->   SELECT ROW_NUMBER() OVER (ORDER BY AVG(x))
    ->   FROM (SELECT 1 AS x WHERE 1=0) AS d WHERE x
    -> );
+---+
| 1 |
+---+
| 1 |
+---+

No crash. UNION/EXCEPT/INTERSECT variants also work correctly.

Basing the PR against the correct MariaDB version

• This is a bug fix and the PR is based against the latest MariaDB development branch.

PR quality check

• I have checked the CODING_STANDARDS.md file and my PR conforms to this where appropriate.
• For any trivial modifications to the PR, I am fine with the reviewer making the changes themselves.

[gemini-code-assist]

Code Review

This pull request fixes a crash in execute_degenerate_jtbm_semi_join (MDEV-39918) by restricting the degenerate JTBM semi-join path to subqueries using SINGLE_SELECT_ENGINE. Other engine types, such as UNION_ENGINE (used for UNION, EXCEPT, INTERSECT, and recursive CTEs), are now correctly routed through normal JTBM materialization. Test cases covering these scenarios have also been added. There are no review comments, and I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gkodinov]

Thank you for your contribution! This is a preliminary review.

Since this is a bug fix, can you please rebase it on 10.11?

Otherwise it looks OK to me.