Skip to content

[feature](external) Add generic connector write framework prerequisites - #68320

Open
suxiaogang223 wants to merge 6 commits into
apache:masterfrom
suxiaogang223:codex/paimon-write-framework
Open

[feature](external) Add generic connector write framework prerequisites#68320
suxiaogang223 wants to merge 6 commits into
apache:masterfrom
suxiaogang223:codex/paimon-write-framework

Conversation

@suxiaogang223

@suxiaogang223 suxiaogang223 commented Sep 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: #65086

Related PR: #67395

Problem Summary:

Forward-porting Paimon table writes exposes four gaps in the generic connector write framework. Implementing them as Paimon-specific branches in FE Core would duplicate the existing write pipeline and would also cause Paimon row-level changes to be planned with Iceberg position-delete semantics.

This PR adds the missing connector-neutral framework capabilities:

  1. Write distribution. A connector can describe routing columns, an opaque partition-function name and options, and writer-assignment semantics. FE Core only resolves columns and transports the description; BE performs the routing. Unsupported functions and unsupported BE execution versions fail explicitly. The existing reserved BE execution version 13 is reused, with no Paimon-specific execution version.
  2. Row-level DML. Connectors declare whether changes use file-position deletes or changelog rows. Iceberg retains its position-delete plan. The generic changelog plan converts DELETE, UPDATE, MERGE, and DELETE USING into an operation marker plus the output row, while the connector supplies the marker encoding, primary keys, validation, and label prefix. Unsupported representations fail instead of falling through to an unrelated write path.
  3. Commit reports. BE can return opaque connector commit fragments through the existing external-write report lifecycle. FE acknowledges only accepted reports, retries remain safe after a lost response, and duplicate reports do not add commit data twice. FE Core does not interpret connector-owned bytes.
  4. Variant writes. FE type checking/construction and BE Arrow serialization support Binary Variant V2 as a generic capability required by connector writes.

The implementation intentionally does not add Paimon-specific planner classes, hidden-column names, Thrift structures, or a new write system. Paimon-specific table rules, fixed-bucket hash implementation, writer, and transaction interpretation remain in the Paimon integration that will follow this prerequisite PR.

The connector SPI surface changes once, so connector.plugin.api.version is increased from 10 to 11 once for the whole PR.

Release note

None

Check List (For Author)

  • Test:
    • Targeted FE unit tests for connector SPI, row-level DML planning, commit report acknowledgement/serialization, Variant type handling, and connector write distribution
    • Relevant FE Maven reactors compiled successfully while running the targeted unit tests
    • BE formatting with clang-format 16 and build-support/check-format.sh
    • build-support/check-build-hygiene.sh
    • BE compilation and BE unit tests were not run for the latest distribution change
  • Behavior changed: Yes. Connectors can declare row-change representation and write distribution, return opaque acknowledged commit data, and write Binary Variant V2 through the shared framework.
  • Does this need documentation: No

@hello-stephen

hello-stephen commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
suxiaogang223 force-pushed the codex/paimon-write-framework branch from fc95401 to 4b96e9f Compare September 21, 2026 07:29
@suxiaogang223
[feature](fe) Add opaque connector commit report channel
bfa94a4 
### What problem does this PR solve?

Issue Number: apache#65086

Related PR: apache#67395

Problem Summary: External-file commit reports only carry source-specific Thrift structs. A new connector would need branches in FE core and could miss the existing acknowledgement and retry-safe ownership transfer. Add an opaque binary commit-data field, share the BE report-size budget with Iceberg, and feed the bytes into the owning connector transaction through both coordinator paths. Keep the existing Hive, Iceberg, and MaxCompute fields unchanged.

### Release note

None

### Check List (For Author)

- Test: FE build with ./build.sh --fe -j 4; FE unit tests with ./run-fe-ut.sh --run org.apache.doris.qe.QeProcessorImplReportAckTest,org.apache.doris.transaction.CommitDataSerializerTest; BE clang-format 16 check and header hygiene check. BE compile and unit tests not run.
- Behavior changed: Yes (new opt-in connector commit-data report channel; existing connector paths unchanged)
- Does this need documentation: No
@suxiaogang223
[fix](fe) Route connector row-level writes by representation
06ddfc3 
### What problem does this PR solve?

Issue Number: apache#65086

Related PR: apache#67395

Problem Summary: The shared row-level DML registry selected the Iceberg transform solely from DELETE or MERGE capability. A connector such as Paimon can support the same operations while encoding changes as changelog rows, which would incorrectly route it through Iceberg position-delete planning. Add an engine-neutral row-change representation to the connector SPI, declare Iceberg as position-delete, select its transform by both representation and operation, and fail loudly when a connector advertises row-level writes before its representation has a registered plan.

The SPI surface changes, so bump the connector plugin API major and update the recorded surface in the same commit.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - Connector SPI test suite
    - IcebergRowLevelDmlTransformTest and PluginDrivenExternalTableTest
    - IcebergWritePlanProviderTest#declaresFullWriteOperationSet
    - FE-core, connector SPI, and Iceberg checkstyle
- Behavior changed: Yes (connector row-level DML is selected by its declared representation)
- Does this need documentation: No
@suxiaogang223
suxiaogang223 force-pushed the codex/paimon-write-framework branch from 4b96e9f to 06ddfc3 Compare September 21, 2026 07:31
@suxiaogang223
[fix](fe) Complete generic connector commit report handling
c7dd526 
### What problem does this PR solve?

Issue Number: apache#65086

Related PR: apache#67395

Problem Summary: Generic connector commit payloads were acknowledged by query execution but were not handled consistently by all execution report paths. Centralize payload detection and dispatch so coordinator, load, and job reports all deliver opaque connector commit data to the transaction.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - ./run-fe-ut.sh --run CommitDataSerializerTest,AbstractJobProcessorTest,QeProcessorImplReportAckTest
- Behavior changed: No
- Does this need documentation: No
@suxiaogang223
[feature](fe) Add generic connector changelog DML planning
14d8be7 
### What problem does this PR solve?

Issue Number: apache#65086

Related PR: apache#67395

Problem Summary: The master row-level DML framework only models position-delete sinks and carries Iceberg-specific routing details in FE Core. Add a connector-neutral changelog row representation for DELETE, UPDATE, and MERGE, move connector-owned encoding and validation behind the write SPI, and generalize the existing position-delete transform. Preserve required short-circuit evaluation for MERGE branch expressions and track the operation-column row shape explicitly through connector sinks.

### Release note

Add connector-neutral framework support for changelog row-level writes.

### Check List (For Author)

- Test: Unit Test
    - run-fe-ut.sh --run ConnectorPluginSurfaceTest,ConnectorRowLevelDmlContractTest,ConnectorChangelogPlanBuilderTest,PositionDeleteRowLevelDmlTransformTest,PhysicalConnectorTableSinkTest,CaseWhenToCompoundPredicateTest
- Behavior changed: Yes. Connector row-level DML is routed by connector-declared row representation and capabilities.
- Does this need documentation: No
@suxiaogang223
[feature](variant) Add Binary Variant V2 write support
c3dd8d0 
### What problem does this PR solve?

Issue Number: apache#65086

Related PR: apache#67395

Problem Summary: Master lacks the generic Binary Variant V2 Arrow transport and nested constructor support needed by connector writes. Add generic binary-struct serialization and connector target casts while preserving master's Variant property compatibility rules.

### Release note

Support Binary Variant V2 serialization for connector writes.

### Check List (For Author)

- Test:
    - FE Unit Test: `./run-fe-ut.sh --run CheckCastTest,FunctionRegistryTest`
    - BE format check: `build-support/clang-format.sh` and `build-support/check-format.sh`
- Behavior changed: Yes. Connector writes can serialize Binary Variant V2 values and construct nested Variant values.
- Does this need documentation: No
@suxiaogang223
[feature](external) Add generic connector write distribution
2d0d2e8 
### What problem does this PR solve?

Issue Number: apache#65086

Related PR: apache#66685

Problem Summary: Connector-specific bucket routing was previously modeled with data-source classes and enums in FE Core. Add a connector-neutral write-distribution SPI and an opaque external partition-function transport. FE Core now resolves route columns and forwards the function name and options without knowing a connector, while BE provides the common writer-assignment framework and validates supported functions.

### Release note

Add a generic external table writer distribution framework.

### Check List (For Author)

- Test:
    - FE Unit Test: `./run-fe-ut.sh --run ConnectorPluginSurfaceTest,ConnectorWriteDistributionTest,PhysicalConnectorTableSinkTest`
    - BE format check: `build-support/clang-format.sh` and `build-support/check-format.sh`
    - BE build hygiene: `build-support/check-build-hygiene.sh`
- Behavior changed: Yes. Connectors can request opaque external writer routing without adding data-source code to FE Core.
- Does this need documentation: No
@suxiaogang223 suxiaogang223 changed the title [feature](fe) Add connector write framework prerequisites [feature](external) Add generic connector write framework prerequisites Sep 21, 2026
@suxiaogang223

suxiaogang223 commented Sep 21, 2026

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

hello-stephen commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.54% (2064/2662)
Line Coverage 65.79% (37952/57686)
Region Coverage 53.22% (35560/66816)
Branch Coverage 56.60% (11428/20190)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929 924060929 Awaiting requested review from 924060929 924060929 is a code owner
@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs is a code owner
@morrySnow morrySnow Awaiting requested review from morrySnow morrySnow is a code owner
@englefly englefly Awaiting requested review from englefly englefly is a code owner
@starocean999 starocean999 Awaiting requested review from starocean999 starocean999 is a code owner
@yujun777 yujun777 Awaiting requested review from yujun777 yujun777 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@suxiaogang223 @hello-stephen