You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "hbgstc123 (via GitHub)" <gi...@apache.org> on 2026/07/11 17:53:45 UTC

[PR] [core] Refactor sort compact to produce COMPACT commits [paimon]

hbgstc123 opened a new pull request, #8566:
URL: https://github.com/apache/paimon/pull/8566

   ### Purpose
   
   Follow up on [#7595](https://github.com/apache/paimon/pull/7595) per [this review comment](https://github.com/apache/paimon/pull/7595#issuecomment-4814778161): sort compact should not be modeled as an OVERWRITE commit with extra base-snapshot handling. Instead, it should produce a normal COMPACT commit so the existing commit protocol can perform compact validation and conflict detection.
   
   This PR fixes the concurrent-write data loss risk in sort compact by refactoring Flink/Spark sort compact to:
   
   1. Write sorted output in write-only mode.
   2. Rewrite the written append commit messages into CompactIncrement messages.
   3. Commit them through the normal compact commit path.
   
   No new overwritePartition(baseSnapshotId) APIs or special overwrite conflict logic are introduced.
   
   ### Tests
   
   Core
   
   - SortCompactCommitMessageRewriterTest
     - rewrite to compact messages
     - multi-bucket / cross-bucket rewrite
     - deletion-vector cleanup
     - hash-dynamic hash-index cleanup
     - snapshot sequence range preservation
     - inline compaction rejection
     - compact commit success detection, including expired snapshot gaps
   - ConflictDetectionTest
   - KeyValueTableReadSequenceTest
   - IncrementalDiffSplitReadTest
   - PrimaryKeyTableRawFileSplitReadProviderTest
   
   Flink
   
   - SortCompactCommitterTest
   - SortCompactSinkWriteTest
   - SortCompactActionForAppendTableITCase
     - latest snapshot kind is COMPACT
     - concurrent append between read and commit does not lose data
   - SortCompactActionForDynamicBucketITCase
   - CommitterOperatorTest
   - FlinkSourceBuilderTest
   
   Spark
   
   - SortCompactSparkCommitTest
   - CompactProcedureTestBase updated to expect COMPACT


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Refactor sort compact to produce COMPACT commits [paimon]

Posted by "hbgstc123 (via GitHub)" <gi...@apache.org>.
hbgstc123 closed pull request #8566: [core] Refactor sort compact to produce COMPACT commits
URL: https://github.com/apache/paimon/pull/8566


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org