[Enhancement] Support Paimon 2.0 table writes on master

[damokelis]

Search before asking

I searched existing issues and PRs. #65086 tracks general Paimon write support, and #65868 implemented the Java/JNI write path for branch-4.1. This issue focuses on bringing that capability to the current master architecture and upgrading the shared runtime to Apache Paimon 2.0.

Description

Current master can read Paimon tables through the new Connector SPI, but it still uses Paimon 1.3.1 and has no complete external-table write path. The implementation merged into branch-4.1 cannot be applied directly because master has moved catalog, transaction, planning, and scan/write contracts into the new connector modules.

The desired behavior is:

• use one Paimon 2.0 runtime version across FE serialization, BE JNI, scanner, and writer;
• support CREATE TABLE, DROP TABLE, INSERT INTO, INSERT OVERWRITE, and CTAS against Paimon external catalogs;
• support filesystem, HMS, REST, HDFS, and S3-compatible object storage catalogs;
• support append-only and primary-key tables, partitioned writes, and Paimon bucket assignment;
• preserve Doris statement atomicity with FE-coordinated commit, multi-BE commit aggregation, idempotent retry/reconciliation, and abort cleanup;
• map Doris primitive and complex types, including Paimon 2.0 BLOB plus Doris VARIANT and VECTOR encodings;
• keep read compatibility for existing Paimon tables after the runtime upgrade.

Proposed solution

Port the proven Java SDK/JNI design from #65868 to the current Connector SPI instead of restoring the old datasource implementation:

ConnectorWritePlanProvider
  -> PaimonTableSinkOperator
  -> PaimonTableWriter
  -> JNI Paimon writer
  -> Paimon CommitMessage
  -> FE ConnectorTransaction

The writer should delegate partition routing, bucket/index assignment, schema defaults, file creation, and commit semantics to the Paimon 2.0 SDK. Doris should own query lifecycle, memory limits, commit-message transport, acknowledgement, retry reconciliation, and cleanup.

BLOB and VECTOR can be represented using Paimon 2.0 physical types plus stable column directives. VARIANT should use a lossless Arrow struct carrying value and metadata binaries so nested and typed values survive round trips.

Validation

The change should include:

• Java unit tests for Arrow conversion, schema mapping, commit codec, JNI writer, connector write planning, and transaction handling;
• C++ unit tests for VARIANT transport, JNI schema metadata, report budgets, acknowledgement, and cleanup;
• regression coverage for create/drop, insert, overwrite, CTAS atomicity, and BLOB/VARIANT/VECTOR;
• live filesystem/HMS/REST/S3-compatible catalog tests against Paimon 2.0 data.

Are you willing to submit PR?

Yes. A draft implementation for master is being validated and will reference this issue.

Code of Conduct

I agree to follow the Apache Software Foundation Code of Conduct.

[damokelis]

Implementation and live-validation update (2026-08-15):

• Updated draft PR [feature](paimon) Add Paimon 2.0 transactional writes #66790 to head 01d75c27da on current master.
• FE Paimon connector: 552 tests passed, 1 skipped; focused C++ tests: 13/13 passed.
• Linux release BE build, changed-line clang-tidy, clang-format, checkstyle, and diff checks passed.
• Deployed the new BE to the TKE test cluster and verified the backend remained alive with the same backend ID.
• Verified COS Parquet+VARIANT reads with automatic JNI fallback (SplitStat [type=JNI]) and verified Vortex+VARIANT is rejected before remote table creation with no residue.
• Both continuous DTS-to-Fluss pipelines and Fluss-to-Paimon tiering remained healthy; queues drained to zero and tiering checkpoints continued advancing.
• Exact counts succeeded for 28 of 31 COS tables. Three heavily fragmented tables remain blocked by external COS getFileStatus API timeouts even after increasing the S3A request timeout from 15s to 120s; their Paimon snapshot and file metadata are readable. This is recorded explicitly in the PR instead of being reported as a successful full sweep.

The PR remains draft. The repository's automated /review workflow only accepts triggers from a MEMBER, OWNER, or COLLABORATOR.