Skip to content

[Tracking] Protocol-agnostic session and execution layer: MySQL and Arrow Flight SQL as equal front ends #67577

Open
Open
[Tracking] Protocol-agnostic session and execution layer: MySQL and Arrow Flight SQL as equal front ends#67577
Labels
tracking issue

Description

@morningman
morningman
opened on Sep 7, 2026

This issue tracks the work that makes MySQL and Arrow Flight SQL equal front ends over one session
layer in the FE: one session object, one result-encoding abstraction, one Doris-to-Arrow type
mapping, one connection pool and quota, and one prepared-statement registry.

Part of #65615. The design is being written up as a DSIP; this issue is the implementation tracker.
If you hit one of the problems listed here, please open a separate bug report and link it back.

#67578 builds on this one: completing the Arrow Flight SQL server and shipping an official Doris
ADBC driver needs the session lifecycle, the result path and the prepared-statement registry that
this issue unifies. The two can be discussed in parallel, but that work only lands cleanly on top
of what is tracked here.

Relationship to the earlier protocol SPI work. #60355 / #60361 (both closed) decoupled the
listeners: how a protocol server is discovered, configured and started. This issue covers
everything behind the listener - session state, the result path, type mapping, connection governance
and prepared statements - which is still written against the MySQL protocol and special-cased for
Arrow Flight SQL.

Why

An Arrow Flight SQL session already is a ConnectContext, SHOW PROCESSLIST lists it and KILL
works on it. What is not shared is most of what the execution layer does afterwards. A read of the
FE session/connection/result paths and the BE result sinks turned up the following; items with a PR
have been verified locally, the rest are code-reading conclusions and are marked as such.

  • The execution layer branches on the protocol in roughly 40 places (StmtExecutor 18,
    ConnectProcessor 8, nine Commands, plus Coordinator, NereidsCoordinator,
    StatementContext, FrontendServiceImpl) and reaches for MysqlChannel in roughly 50. A Flight
    session throws from getMysqlChannel(), so every unguarded reach is a live failure rather than a
    style problem - see [fix](arrow-flight) Forward a statement to the master FE without touching the MySQL channel #67569 for one that reached production.
  • There are three Doris-to-Arrow type mappings: the BE's convert_to_arrow_type, the FE's
    FlightSqlSchemaHelper.getArrowType (used by CommandGetTables), and FlightSqlChannel, which
    encodes every FE-side result as utf8. The FE mirror already disagrees with the BE in four types.
  • Flight connections are governed by their own pool with their own limit
    (arrow_flight_max_connections) plus a per-user token LRU, so qe_max_connection and
    max_user_connections do not apply to them, and a live session that the LRU evicts surfaces to the
    client as invalid bearer token.
  • Flight prepared statements do not use the FE's prepared-statement machinery:
    createPreparedStatement stores the SQL text and returns a placeholder schema, and parameter
    binding is not implemented.

Status convention

Same as #65615: [x] means merged or confirmed complete; [ ] means open or needs follow-up.

Stage 0 - independent bug fixes

Defects found by the survey that do not depend on the refactor. All three are merged and carry
dev/4.2.x, so they reach the 4.2 branch ahead of the refactor.

Stage 1 - one session, one result path

Behavior-preserving refactor, guarded by golden tests that pin the MySQL packet bytes. Complete:
all five PRs merged between 2026-09-11 and 2026-09-14, and the performance baseline shows no change.

  • Golden tests first: [test](protocol) Record a golden baseline of MySQL packets and Arrow Flight results #67789, merged 2026-09-11. A recording MysqlChannel drives a statement set through
    MysqlConnectProcessor/StmtExecutor and the produced packets are compared byte for byte; the same
    statements then run through the Flight processor and their schema and rows are compared. This is the
    safety net for everything below, so it lands before any refactor. 27 MySQL cases and 8 Flight
    statements, recorded as annotated hexdumps rather than binary blobs so a diff is reviewable. Two
    responses are recorded as shape rather than bytes, because their payload moves with unrelated
    changes: a parser error carries the grammar's whole keyword list, and an EXPLAIN carries the
    current plan text. COM_STMT_EXECUTE is out of scope for the golden files because
    supportHandleByFe() is false for it, so its result always comes from a BE - the cursor-fetch
    packet boundaries stay covered by prepared_stmt_p0 instead.
  • ProtocolAdapter: [refactor](session) Bind a ConnectContext to one ProtocolAdapter instead of a Flight subclass #67835, merged 2026-09-11. The per-protocol half of ConnectContext
    (channel, capabilities, accept loop, close, the COM_STMT_EXECUTE state, the Flight result cache,
    endpoints and deferred executors) moves behind an interface with a MySQL and a Flight
    implementation; ConnectContext.forMysql / forMysqlProxy / forFlight create the bound
    context. ConnectContext keeps its class name and getter signatures - it is referenced by ~900
    files - and FlightSqlConnectContext goes away. Flight commands get a per-session lock:
    ConnectContext is not thread safe and Flight did not serialize a session's calls. Verified
    against the [test](protocol) Record a golden baseline of MySQL packets and Arrow Flight results #67789 baseline: not a byte changed. The internal (no-client) context stays a MySQL
    context over a DummyMysqlChannel for now, because the execution layer still keys its result
    path on ConnectType; a distinct internal adapter follows once those branches are gone.
  • Package move: [refactor](arrow-flight) Move Arrow Flight SQL out of the service package into org.apache.doris.arrowflight #67866, merged 2026-09-11. org.apache.doris.service.arrowflight becomes
    org.apache.doris.arrowflight. Flight was filed under service in 2023 ([feature-wip](arrow-flight)(step3) Support authentication and user session #24772) next to the thrift
    FrontendServiceImpl; it is now a peer of the top-level mysql package, and [refactor](session) Bind a ConnectContext to one ProtocolAdapter instead of a Flight subclass #67835 already had to
    mirror a protocol sub-package on both sides. A pure git mv (36 files, only package/import
    lines change, nothing references the old name by string), sequenced before ResultSender so the
    new implementations land in the final place.
  • ResultSender: [refactor](session) Move result encoding behind ResultSender #67883, merged 2026-09-12. The result-encoding half moves out of
    StmtExecutor/ConnectProcessor: qe.protocol.ResultSender with MysqlResultSender /
    FlightResultSender, StmtExecutor without a serializer field or MysqlChannel parameters, the
    three text-result paths (EXPLAIN, EXPLAIN PLAN PROCESS, REPLAY) through the one
    sendResultSet, and ConnectProcessor without its connectType field (the per-statement
    protocol work of a multi-statement request is adapter.finishStatement). The MySQL golden is
    byte-identical; the Flight golden changes in exactly one entry, EXPLAIN PLAN PROCESS, which
    had no Flight branch before and now returns its rows. Also carries the leftovers of the [refactor](session) Bind a ConnectContext to one ProtocolAdapter instead of a Flight subclass #67835
    review (lock wait bound = getExecTimeoutS(), FlightRuntimeException passed through the
    producer's catch-alls, a WARN when the lock is given up).
  • Remaining protocol branches in the execution layer go to zero: [refactor](session) Replace the execution layer's ConnectType branches with protocol capabilities #67900, merged 2026-09-14.
    StmtExecutor, ConnectContext.supportHandleByFe, the nine insert / transaction Commands, both
    coordinators, the short-circuit rule and FEOpExecutor stop asking getConnectType();
    ProtocolAdapter gains one method per use: canReplayForwardedQueryResult, supportsFeSideResult,
    supportsShortCircuitPointQuery, canRetryQuery, the lifecycle hooks beforeStatement /
    beforeAttempt / beforeQuery / returnsResultFromLocal, and fillForwardRequest (with
    MysqlProtocolAdapter.restoreFromForwardRequest as the master's side). The Flight-only
    returnResultFromLocal flag is no longer flipped from outside the adapter, and FlightResultSender
    no longer has to undo a flip for EXPLAIN. The behavior changes, all recorded in the MySQL golden
    (28 -> 33 cases; the Flight golden is byte-identical): the channel is reset once, when a statement
    starts, instead of in the query path and in nine commands, so a request from a client without
    CLIENT_MULTI_STATEMENTS whose last statement is not a query (select 1; set @a = 1) delivers
    only the last response instead of the buffered result set of the SELECT followed by an OK -- a
    stream no client parses; MysqlChannel.reset() now also rewinds the sequence id to the last packet
    the client saw, so such a response (select 1; select 2, select 1; select * from no_such_table)
    starts at sequence id 1 instead of leaving the hole pymysql and libmysqlclient rejected (the
    pre-existing bug noted on 2026-09-13, fixed here because the reset-per-statement had widened it);
    ProxyMysqlChannel drops a failed attempt's packets on reset, so a forwarded query the master
    retries no longer delivers both attempts' packets; and a Flight query replanned after a cloud
    NEED_REPLAN error starts every attempt through beforeAttempt, which withdraws exactly the
    endpoints the failed attempt registered. The recording channel of the golden models the send buffer
    and the sequence rewind, so the golden shows what reaches the client. Wired end to end by
    ProtocolCapabilityWiringTest (forward refusal, retry, an internal executor answering on the
    caller's connection) and three new regression suites: test_arrow_flight_session_lifecycle,
    test_multi_statement_response (a raw MySQL-protocol client covers all four
    CLIENT_MULTI_STATEMENTS x CLIENT_DEPRECATE_EOF combinations) and the two-FE
    test_mysql_forward_to_master.
  • Performance baseline, run 2026-09-13: three FE builds - before Stage 1 (60042611fea), after
    the package move (af333cc583d) and the [refactor](session) Replace the execution layer's ConnectType branches with protocol capabilities #67900 branch - against the same BE, data and metadata.
    A 10M-row wide table drained over mysql --quick and over a raw socket, select 1 and one-row BE
    queries over Connector/J at 1 and 8 threads, ADBC fetch_arrow_table / select 1 /
    show variables. Every difference is within the run-to-run noise of a single build (FE-side paths
    within +/-1%; the few cells above 3% changed sign on interleaved re-runs).

#67520 (merged 2026-09-10) lands in the middle of this surface: it touches ConnectProcessor,
MysqlConnectProcessor, StmtExecutor, FEOpExecutor, MysqlProto and ConnectContext, and adds
MysqlCursorFetchCompatibility, a client-capability branch that decides where a cursor result's
terminators go. The adapter extraction absorbs it rather than works around it: that logic belongs to
the MySQL adapter (#67835 moved it there as MysqlProtocolAdapter.clientConsumesCursorMetadataTerminator).
#67753 (merged 2026-09-11, right after #67835) adds one more thing the ResultSender step has to
carry: an internal query run on behalf of a client (the IVM dry run) streams its rows to that
client's MySQL channel, so StmtExecutor grew channel-taking overloads of its send methods. The
result sender is handed over by the caller in that case rather than taken from the executor's own
session.

Stage 2 - one type mapping

Deferred (2026-09-14). The BE Arrow type layer is being refactored separately - the refactor
#67530 was told to wait for - and changing the mapping now would collide with it, so #67530 and
#65789 were closed unmerged and this stage lands after that refactor, on top of its mapping. Until
then every new interface uses the current mapping exactly as it is, through one FE entry point
(FlightSqlSchemaHelper.getArrowType extracted into DorisArrowTypeMapping with no value
changed), and the known-wrong cells are kept deliberately rather than fixed piecemeal: GetTables
reports TIMESTAMPTZ with a fixed UTC zone and Null for TIMEV2 / VARBINARY / AGG_STATE;
AGG_STATE travels as utf8; doris_type metadata covers only top-level LARGEINT / IPV4 / IPV6, no
ARROW:extension:name anywhere; FE-side results are all utf8; the oversized-column large_utf8
mismatch of #65789 stays. The server-side items of #67578 Part A (session actions, metadata
commands, prepared statements, cancellation, parallel endpoints, ingest) go first instead, with
Stage 3 between the first two, and the BE-touching ones stay clear of the type layer.

  • The one FE entry point first, done now: FlightSqlSchemaHelper.getArrowType and the field
    builder it feeds extracted into org.apache.doris.arrow.DorisArrowTypeMapping with no value changed,
    the mapping pinned as it stands by a table-driven unit test (one row per PrimitiveType, the
    known-wrong cells included and marked) and by a regression suite that reads the GetTables schema
    of a table of every type through a raw Flight SQL client. [refactor](arrow-flight) Extract the Doris-to-Arrow type mapping into DorisArrowTypeMapping #68315, opened 2026-09-21.
  • One enumerable Doris-to-Arrow mapping, shared by a table-driven BE unit test and an FE test that
    validates the FE mirror against the same golden file, and the four known FE/BE disagreements fixed.
    [feature](timestamp_ns) Add end-to-end TIMESTAMP_NS support #66761 added TIMESTAMP_NS to both mappings separately - one more line in arrow_row_batch.cpp and
    one in FlightSqlSchemaHelper - which is exactly the drift this stage removes; the golden file
    covers it.
  • Field metadata completed on the BE: doris_type for every non-native type plus
    ARROW:extension:name (doris.largeint, doris.ipv4, doris.ipv6, doris.bitmap, doris.hll,
    doris.quantile_state, doris.agg_state, doris.time), with JSONB and VARIANT using the canonical
    arrow.json extension. Storage types do not change: convert_to_arrow_type also serves the
    Spark and Flink connectors' read path and Python UDFs, so changing them would change those
    protocols. In particular LARGEINT stays a decimal string rather than becoming decimal128(38,0),
    which cannot represent an int128 (39 decimal digits; Arrow caps decimal128 precision at 38), and
    IPV4/IPV6 stay int32/utf8 rather than fixed_size_binary.
  • FE-side results (SHOW, EXPLAIN, replayed proxy results) built as typed vectors instead of
    all-utf8.
  • A cross-protocol regression suite: the same query over JDBC and over Flight, compared column by
    column across the whole type matrix.

Stage 3 - one connection pool and quota

Builds on #67504 (merged 2026-09-07), which stopped a finished Flight query from holding its
coordinator until wait_timeout.

Stage 4 - prepared statements

  • The prepared-statement registry is extracted from the MySQL COM_STMT_* handling so both front
    ends share it.
  • Flight createPreparedStatement/getFlightInfoPreparedStatement/closePreparedStatement go
    through it and return the real result and parameter schemas.
  • acceptPutPreparedStatementQuery binds Arrow parameter batches, so Flight SQL JDBC with
    useServerPrepStmts=true and the ADBC default path work.

Known gaps, not yet scheduled

Code-reading conclusions from the same survey. They are not blocking the stages above; each needs its
own report or PR.

  • A forwarded statement's result set is dropped over Flight. Even after [fix](arrow-flight) Forward a statement to the master FE without touching the MySQL channel #67569, a statement
    that is forwarded to the master and returns rows gives the Flight client nothing, because the proxy
    result is replayed by ConnectProcessor.finalizeCommand(), which is MySQL-only. Affects only
    statements that both need forwarding and carry a result set.
  • arrow_flight_token_alive_time_second is applied in the wrong unit. createToken() converts
    the configured seconds as if they were minutes, so the recorded expiry is 60x too far out; the token
    is actually evicted by the Guava cache's own expiry, which makes the explicit expiry check dead code.
    Gone with [refactor](arrow-flight) Make the bearer token the credential of exactly one session #68266 (merged 2026-09-21): the setting is a deprecated no-op and FlightTokenDetails is
    deleted.
  • AGG_STATE is advertised as utf8 while carrying non-UTF-8 bytes - the same shape as the
    Iceberg BINARY problem in [Bug] Arrow Flight SQL: Iceberg BINARY column is declared as Arrow string but carries invalid UTF-8 #67371.
  • select 1 and select @@var always go to the BE on a Flight session, because
    supportHandleByFe() is hard-coded false for Arrow Flight SQL. Enabling FE-side results for Flight
    is deliberately out of scope until the FE result path is typed (Stage 2).
  • SHOW TABLES / DATABASES / COLUMNS / TABLE STATUS ... WHERE hang on a Flight session. The
    four commands run an internal query on the session's own context, so the plan gets an Arrow result
    sink while the frontend pulls the rows over fetch_data; the request waits until it times out. The
    LIKE form takes the metadata path and is unaffected. Found while writing [refactor](session) Replace the execution layer's ConnectType branches with protocol capabilities #67900, which
    deliberately leaves that path as it is (the coordinators keep deciding on the per-statement flag,
    not on a protocol capability).
  • The master answers a forwarded query with statusCode 1105. ConnectProcessor.proxyExecute
    reports 0 only when the state is OK; a query ends in EOF, so the follower sees a failure code
    with a null message and its multi-statement loop stops after the first statement: select 1; select 2 on a follower with force_forward_all_queries runs only select 1.
  • The handshake does not advertise CLIENT_MULTI_STATEMENTS / CLIENT_MULTI_RESULTS.
    Connector/J only asks for CLIENT_MULTI_STATEMENTS when the server advertised it, so a JDBC client
    with allowMultiQueries=true never receives the intermediate responses of a multi-statement
    request, although the server side handles them (the golden shows it, and the mysql CLI, which sets
    the flag regardless, gets them). test_multi_statement_response reaches that path with a raw
    protocol client instead.

Upstream PRs this work depends on

Progress

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions