You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Stephen0421 (via GitHub)" <gi...@apache.org> on 2026/07/23 10:09:18 UTC
[PR] [core] Split chain table batch scans by key range for read parallelism. [paimon]
Stephen0421 opened a new pull request, #8831:
URL: https://github.com/apache/paimon/pull/8831
### Purpose
Chain table batch reads currently produce one split per bucket, which limits read parallelism for large buckets. This PR adds key-range split generation for chain table batch scans:
- Introduces `chain-table.split.key-range-enabled` (default: `true`); set to `false` to fall back to one split per bucket
- Batch scans use `IntervalPartition` + `BinPacking` to split snapshot and delta files by key range; files with overlapping key ranges always stay in the same split so all versions of a key across branches are merged correctly
- Streaming reads are unchanged and still use the original one-split-per-bucket behavior
### Tests
- Added `ChainTableUtilsTest.testBuildChainSplitsWithKeyRangeSplitting`: verifies key-disjoint sections are split into separate splits and overlapping snapshot/delta files are kept together
- Added `ChainTableUtilsTest.testBuildChainSplitsKeyRangeSplittingLargeTargetKeepsOneSplit`: verifies a large `targetSplitSize` packs all sections into a single split
- Added `ChainTableUtilsTest.testBuildChainSplitsWithoutKeyRangeSplitting`: verifies a null `keyComparator` preserves the original one-split-per-bucket behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Split chain table batch scans by key range for read parallelism. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8831:
URL: https://github.com/apache/paimon/pull/8831#issuecomment-5058694891
After adding `chain-table.split.key-range-enabled`, `docs/generated/core_configuration.html` was not updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Split chain table batch scans by key range for read parallelism. [paimon]
Posted by "Stephen0421 (via GitHub)" <gi...@apache.org>.
Stephen0421 commented on PR #8831:
URL: https://github.com/apache/paimon/pull/8831#issuecomment-5058777688
> After adding `chain-table.split.key-range-enabled`, `docs/generated/core_configuration.html` was not updated.
OK, fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Split chain table batch scans by key range for read parallelism. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8831:
URL: https://github.com/apache/paimon/pull/8831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org