You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jordepic (via GitHub)" <gi...@apache.org> on 2026/09/12 14:44:29 UTC

[PR] [core] Allow custom primary-key compaction rewriters [paimon]

jordepic opened a new pull request, #9762:
URL: https://github.com/apache/paimon/pull/9762

   ### Purpose
   
   Primary-key compaction chooses its rewriter inside a private factory. An external engine currently has to reconstruct the writer setup to replace the file read/merge/write work with its own implementation.
   
   Add `TableWrite.withCompactRewriterFactory(...)`, backed by the existing `CompactRewriter` boundary. Each partition/bucket receives its own rewriter, and the factory receives Paimon's normal rewriter so implementations can delegate unsupported operations. Paimon retains input selection, background scheduling, and commit coordination. This PR adds the extension point; it does not add a native compaction engine.
   
   ```java
   write.withCompactRewriterFactory((partition, bucket, defaultRewriter) ->
           new CustomCompactRewriter(partition, bucket, defaultRewriter));
   ```
   
   The API documents ownership and recovery: the returned rewriter owns the default, failed creation closes the default, and the factory must be installed before bucket creation and again on recovered writers. Write-only mode does not invoke the factory; append, postpone, and primary-key clustering writers reject it. The Java writing guide includes the integration contract.
   
   cc @JingsongLi — feedback on exposing this boundary for external/native engines would be appreciated.
   
   ### Tests
   
   24 focused Core tests passed on JDK 11, with normal Checkstyle, Spotless, and enforcer checks enabled:
   
   ```bash
   mvn -pl paimon-core -am -Pflink1 \
     -DwildcardSuites=none \
     -Dtest=CompactRewriterFactoryTest,MergeTreeCompactManagerFactoryTest,TableWriteTest,KeyValueFileStoreWriteTest \
     -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false test
   ```
   
   Coverage includes real rewrites and metadata upgrades across reopened writers; fallback selection for none/input/lookup/full-compaction and deletion vectors; final table contents; rewriter closure; write-only behavior; unsupported append writers; late installation; and cleanup/suppressed exceptions when factory creation fails.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Allow custom primary-key compaction rewriters [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9762:
URL: https://github.com/apache/paimon/pull/9762


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Allow custom primary-key compaction rewriters [paimon]

Posted by "jordepic (via GitHub)" <gi...@apache.org>.
jordepic commented on PR #9762:
URL: https://github.com/apache/paimon/pull/9762#issuecomment-5646615627

   Thank you! Hoping to use this in https://github.com/datafusion-contrib/StreamFusion/tree/main


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org