You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wayneli-vt (via GitHub)" <gi...@apache.org> on 2026/06/02 06:17:39 UTC
[PR] [spark] Support split-granularity bin packing for data evolution tables [paimon]
wayneli-vt opened a new pull request, #8072:
URL: https://github.com/apache/paimon/pull/8072
### Purpose
Support split-granularity bin packing when `data-evolution.enabled` is enabled.
Data evolution splits must remain intact. Add split-granularity bin packing to avoid reshuffling files across splits, while still grouping whole splits by target size. Oversized splits are kept as-is.
Also refactor test helpers to reduce boilerplate.
### Tests
- Added `BinPackingSplitsTest` coverage for split-granularity packing.
- Added coverage for oversized data evolution splits.
- Existing bin packing tests remain unchanged and passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support split-granularity bin packing for data evolution tables [paimon]
Posted by "wayneli-vt (via GitHub)" <gi...@apache.org>.
wayneli-vt commented on PR #8072:
URL: https://github.com/apache/paimon/pull/8072#issuecomment-4603982626
@JingsongLi Thanks for the thorough review and for verifying the compilation! Glad the split packing fix works for data evolution semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support split-granularity bin packing for data evolution tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8072:
URL: https://github.com/apache/paimon/pull/8072#issuecomment-4603803342
Reviewed the updated split packing logic with focus on Java/Paimon data-evolution semantics.
The important part here is that data-evolution `DataSplit`s are now packed as whole splits instead of being broken into smaller file groups, so the field-bunch / split-level schema-evolution assumptions are preserved. I also verified the common Spark module compiles with:
```shell
mvn -pl paimon-spark/paimon-spark-common -am -Pfast-build -DskipTests compile
```
No additional blocker from my side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support split-granularity bin packing for data evolution tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8072:
URL: https://github.com/apache/paimon/pull/8072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support split-granularity bin packing for data evolution tables [paimon]
Posted by "wayneli-vt (via GitHub)" <gi...@apache.org>.
wayneli-vt commented on PR #8072:
URL: https://github.com/apache/paimon/pull/8072#issuecomment-4600809719
> I found one blocker in the newly added test.
>
> `BinPackingSplitsTest` fails on this PR head:
>
> ```shell
> mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -DfailIfNoTests=false \
> -DwildcardSuites=org.apache.paimon.spark.BinPackingSplitsTest -Dtest=none test
> ```
>
> The failure is in `Paimon: pack data evolution splits by split granularity`:
>
> ```
> expected: <2> but was: <3>
> at BinPackingSplitsTest.scala:90
> ```
>
> The test comment assumes each split is 60B and two splits fit into the 150B target, but `computeMaxSplitBytes` also derives the effective max split size from `SplitUtils.splitSize + fileCount * openCost` and `spark.sql.files.minPartitionNum` / default parallelism. With the current config, the effective max size is smaller than 150B, so every split stays in its own partition.
>
> Could you please either set `spark.sql.files.minPartitionNum -> "1"` in this test or adjust the test sizes/expectation so the suite passes reliably?
Thanks for your review! I updated the test to explicitly set `spark.sql.files.minPartitionNum` to `1`.
Since `computeMaxSplitBytes` prefers `filesMinPartitionNum` over `leafNodeDefaultParallelism`, setting the former makes the expected effective max split size deterministic. I also updated the related configurations in `BinPackingSplitsTest` for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org