You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "sundapeng (via GitHub)" <gi...@apache.org> on 2026/08/11 03:45:17 UTC

[PR] [common] Define provider-neutral FileIO contract [paimon]

sundapeng opened a new pull request, #9161:
URL: https://github.com/apache/paimon/pull/9161

   ## Summary
   
   This PR turns the existing `FileIOBehaviorTestBase` into a provider-neutral FileIO contract and documents the guarantees that Paimon callers may rely on. The contract is shared by LocalFileIO, Hadoop local, HDFS, and S3/MinIO without requiring POSIX directory markers or provider-specific metadata probes.
   
   ## Changes
   
   - Expand the shared contract to 47 tests covering input/output streams, status and listing, delete and mkdir, non-conflicting rename, copy helpers, and two-phase publication and cleanup.
   - Define the same guarantees in the public FileIO, FileStatus, SeekableInputStream, and TwoPhaseOutputStream documentation.
   - Keep provider-dependent cells explicit and unspecified, including missing/file listing inputs, the missing-delete return value, file/directory path collisions, rename conflicts and atomicity, invalid seek offsets, two-phase existing-target behavior, directory-marker layout, and request counts.
   - Align LocalFileIO with the contract by returning immutable status snapshots and making `copyFile(..., false)` reject an existing destination without changing its content.
   - Make renaming-based two-phase discard and cleanup remove only the current writer's staged file.
   - Construct the S3 multipart helper through S3A so Hadoop installs its required upload callbacks.
   
   No new runtime capability or provider profile is introduced. Request-count assertions remain provider-specific work; this contract only defines observable preconditions and postconditions.
   
   ## Testing
   
   - [x] `mvn -pl paimon-common -am -Pfast-build -DwildcardSuites=none -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -Dtest='LocalFileIOBehaviorTest,HadoopLocalFileIOBehaviorTest,RenamingTwoPhaseOutputStreamTest,LocalFileIOTest' test` (106 tests)
   - [x] `mvn -pl paimon-common -am -Pfast-build -DwildcardSuites=none -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -Dtest='HdfsBehaviorTest' test` (51 tests)
   - [x] `DOCKER_HOST=unix:///run/podman/podman.sock TESTCONTAINERS_RYUK_DISABLED=true mvn -pl paimon-filesystems/paimon-s3-impl -am -Pfast-build -DwildcardSuites=none -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -Dtest='S3FileIOTest,S3MultiPartUploadTest' test` (53 tests)
   - [x] `mvn -pl paimon-common,paimon-filesystems/paimon-s3-impl -am -Pfast-build -DskipTests compile`
   - [x] `mvn -pl paimon-common,paimon-filesystems/paimon-s3-impl -am -Pfast-build -DskipTests spotless:check`
   
   The new tests were written before the implementation fixes. On the previous implementation they reproduced the LocalFileIO copy-conflict bug, mutable FileStatus observations, cross-target two-phase discard, and the S3 multipart null-callback failure; all pass after the fixes above.
   
   ## Compatibility
   
   The FileIO method signatures and serialization shape are unchanged. The PR narrows previously ambiguous documentation to the portable behavior already required by Paimon and deliberately leaves storage-specific edge cases outside the common contract.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] 【WIP】[common][filesystem] Add FileIO contract suite [paimon]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@paimon.apache.org>.
github-actions[bot] commented on PR #9161:
URL: https://github.com/apache/paimon/pull/9161#issuecomment-6810433772

   This pull request has had no activity for 90 days and is being marked stale. If you'd like to keep it open, please leave a comment or push a new commit. Thank you for your contribution to Apache Paimon!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org