You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "lilei1128 (via GitHub)" <gi...@apache.org> on 2026/08/14 08:30:15 UTC

[PR] [spark][flink] Support bucket-level compaction for fixed-bucket tables [paimon]

lilei1128 opened a new pull request, #9219:
URL: https://github.com/apache/paimon/pull/9219

   ### Purpose
     For non-partitioned tables with many buckets, or partitioned tables where each partition
     contains many buckets, compacting all buckets in one job may consume excessive planning
     and execution resources and eventually cause OOM errors.
   
     This PR introduces bucket-level compaction for fixed-bucket tables in Flink and Spark.
     Users can split a large compaction into multiple jobs by selecting individual bucket IDs
     or bucket ranges.
   
     For example:
   ```
       CALL sys.compact(
         `table` => 'database.large_bucket_table',
         compact_strategy => 'full',
         buckets => '0-999'
     );
   ```
     A table with many buckets can be compacted in batches such as 0-999, 1000-1999, and 2000-2999.
     
   ### Tests
     - paimon-common/src/test/java/org/apache/paimon/utils/ParameterUtilsTest.java
         - testParseIntegerRanges
         - testInvalidIntegerRanges
   
     - paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/action/CompactActionITCase.java
         - testCompactSpecifiedBucketRangesFromAction
   
     - paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/procedure/CompactProcedureITCase.java
         - testCompactSpecifiedBucketRanges
   
     - paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/procedure/CompactProcedureTestBase.scala
         - Paimon Procedure: compact specified bucket ranges
         - Paimon Procedure: reject buckets for dynamic bucket table
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [spark][flink] Support bucket-level compaction for fixed-bucket tables [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9219:
URL: https://github.com/apache/paimon/pull/9219


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org