Skip to content

【wip】[common][core][s3] Add provider-neutral FileIO contract suite - #9161

Open
sundapeng wants to merge 7 commits into
apache:masterfrom
sundapeng:fileio-contract-suite
Open

【wip】[common][core][s3] Add provider-neutral FileIO contract suite#9161
sundapeng wants to merge 7 commits into
apache:masterfrom
sundapeng:fileio-contract-suite

Conversation

@sundapeng

@sundapeng sundapeng commented Aug 11, 2026

Copy link
Copy Markdown
Member

Summary

This PR adds a provider-neutral, opt-in contract suite for FileIO and applies it to local, Hadoop-local, HDFS, and S3 providers. It follows the layered “Scheme B” design so provider behavior, default-method behavior, API ownership, returned-object behavior, and consumer assumptions are tested independently.

The public FileIO interface, its Javadocs, and the existing FileIOBehaviorTestBase are unchanged.

Changes

  • Add a provider-core contract for streams, status and listing, deletion, rename, copy, UTF-8/atomic helpers, and two-phase output lifecycle.
  • Add request-aware default-method tests backed by a deterministic recording implementation. These verify delegation, overwrite propagation, resource cleanup, lazy listing, atomic temporary-file cleanup, and retry behavior without adding metadata preflights.
  • Classify all 37 public methods declared by FileIO into exactly one owner: provider core, default methods, provider lifecycle, factories, or optional capabilities. Factory selection/discovery and unsupported-capability behavior are covered separately.
  • Cover returned-object contracts, including FileStatus defaults and SeekableInputStream.wrap forwarding and unsupported positioning.
  • Add the test-only StrictContractFileIO consumer guard. It rejects non-portable caller assumptions such as listing non-directories or renaming to an existing destination, and is exercised by catalog, commit/abort, branch, and tag workflows.
  • Extend the existing local, Hadoop-local, HDFS, and MinIO-backed S3 behavior suites from the new opt-in contract base.

Production fixes exposed by the contract

  • LocalFileIO.copyFile(..., false) now reports an existing destination instead of silently succeeding.
  • Local Hadoop tryToWriteAtomic now preserves an existing target and returns false; object-store schemes do not gain this preflight.
  • Renaming two-phase output now discards only its own staged file, preserves concurrently published targets, and performs overwrite fallback only while its staged file still exists.
  • FormatTableCommit rolls back writer-owned UUID targets even when a remote commit publishes and then throws.
  • S3 multipart upload now obtains its WriteOperationHelper from the owning S3AFileSystem, including Hadoop's required multipart callbacks and filesystem context.

Contract boundaries

The suite deliberately leaves provider-specific edge cases unspecified, including ambiguous rename shapes such as a missing source, an existing destination, identical paths, or an invalid destination parent. It also allows missing-file input failure either while opening or on the first read, does not define listing order or pre-close visibility, and does not prescribe exact timestamp precision.

Request-aware tests constrain logical FileIO delegation where it affects behavior, but do not freeze backend-specific HTTP HEAD/LIST request counts or ordering. No provider profile, vendor flag, or new production capability API is introduced.

Testing

  • Passed 243 tests in paimon-common.
  • Passed 119 tests in paimon-core.
  • Passed 6 focused S3 tests.
  • Compiled all 17 filesystem modules in the 22-module Maven reactor.
  • Passed Spotless and diff checks for all changed modules.
  • The full MinIO-backed S3FileIOTest suite was not rerun locally because Docker was unavailable; the pushed PR runs it in CI.

@sundapeng sundapeng changed the title [common] Define provider-neutral FileIO contract [common][core][s3] Add provider-neutral FileIO contract suite Aug 11, 2026
@sundapeng sundapeng changed the title [common][core][s3] Add provider-neutral FileIO contract suite 【wip】[common][core][s3] Add provider-neutral FileIO contract suite Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant