[PR] [core] Support splitting large files in data evolution compaction [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi opened a new pull request, #9709:
URL: https://github.com/apache/paimon/pull/9709

   ### Purpose
   
   Data-evolution compaction skips an oversized file when the file count is below the compaction threshold, and its normal rewrite path produces only one output file. This prevents resizing existing large files after lowering `target-file-size`.
   
   Add `data-evolution.compaction.split-large-files` (default: `false`). When enabled, any normal file strictly larger than twice `target-file-size` can trigger compaction independently of `compaction.min.file-num`, and normal compaction output rolls near the target size. Existing small-file merging remains available; the combined size of multiple versions alone does not bypass the minimum file count.
   
   Preserve row IDs, column sequence metadata, and logical deletions. Redistribute deletion vectors across output files and rewrite associated BLOB/VECTOR files within the new normal-file boundaries, resolving dedicated columns by field ID. Document the option, its additional dedicated-file I/O, and historical snapshot retention.
   
   ### Tests
   
   84 targeted tests passed on JDK 17, including:
   
   - Physical-size threshold boundaries, minimum-file-count behavior, and overflow protection.
   - Historical-file splitting with and without column updates, task serialization, data preservation, and repeated planning.
   - Multiple BLOB columns and a renamed VECTOR column with aligned output ranges and row-by-row value checks.
   - Both deletion-vector bitmap formats, deleted-row positions, row IDs, and existing compaction regressions.
   
   ```sh
   mvn -pl paimon-core -am \
     -DwildcardSuites=none -DfailIfNoTests=false \
     -Dtest=DataEvolutionCompactCoordinatorTest,CompactCandidateRangeCollectorTest,DataEvolutionCompactRangePlannerTest,DataEvolutionNormalCompactTaskTest,DataEvolutionDeletionVectorTest \
     test
   ```
   
   Run without `fast-build`; Checkstyle, Spotless, and Maven Enforcer passed. The existing Byte Buddy agent was preloaded for Mockito in the local sandbox.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Split large normal files in data evolution compaction [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #9709:
URL: https://github.com/apache/paimon/pull/9709


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org