You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "tzphh (via GitHub)" <gi...@apache.org> on 2026/10/09 04:25:38 UTC
[I] [Bug] Data Evolution compaction skips contiguous small files due to size/count thresholds [paimon]
tzphh opened a new issue, #10458:
URL: https://github.com/apache/paimon/issues/10458
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
2.2-SNAPSHOT; upstream master inspected at commit `9c3c5187a4bd045d31b1ed9fb0decd58e6d63e8e` (2026-10-09).
The two planning classes were verified byte-for-byte against this commit, then compiled locally and exercised with synthetic file metadata. This is not a full build or an end-to-end test of that master revision.
### Compute Engine
Java API / paimon-core compaction planning, JDK 17. The issue was investigated in the context of Flink batch compaction, but the reproduction below does not require a Flink cluster.
### Minimal reproduce step
Use a Data Evolution table's normal-file planning path with these inputs:
- One partition, 12 normal Parquet files, each 200 MiB.
- Contiguous, non-overlapping row-ID ranges: file i has firstRowId = i * 100 and rowCount = 100.
- `target-file-size = 512 MB`, `compaction.min.file-num = 5`, `source.split.open-file-cost = 4 MB`.
- No BLOB/VECTOR files, no deletion-vector materialization, large-file splitting disabled.
A minimal metadata-only reproduction can be added to the existing `CompactCandidateRangeCollectorTest` (its current imports already cover this method):
```java
@Test
void testContiguousFilesExceedTargetBelowMinFileCount() {
long mib = 1024L * 1024L;
CompactCandidateRangeCollector collector =
new CompactCandidateRangeCollector(
12, 512 * mib, 1024 * mib, 4 * mib, 5, Long.MAX_VALUE);
for (int i = 0; i < 12; i++) {
collector.add(0, NORMAL_FILE, i * 100L, 100L, 200 * mib);
}
List<String> ranges = new ArrayList<>();
collector.finish((start, end, count) -> ranges.add(start + "-" + end + ":" + count));
assertThat(ranges).isNotEmpty(); // Fails: ranges is empty.
}
```
Run the test from the repository root:
```bash
mvn -pl paimon-core -am -Dtest=CompactCandidateRangeCollectorTest \
-Dsurefire.failIfNoSpecifiedTests=false test
```
A separate synthetic-metadata harness also called `CompactPlanner.compactPlan` directly, bypassing the candidate collector, and then via the collector-selected ranges:
| Input | Target / min files | Candidate ranges | Direct planner tasks | Tasks after candidate filtering |
| --- | --- | ---: | ---: | ---: |
| 12 x 200 MiB | 512 MiB / 5 | 0 | 0 | 0 |
| 12 x 200 MiB | 512 MiB / 2 | 4 | 4 | 4 |
| 12 x 200 MiB | 1024 MiB / 5 | 2 | 2 | 2 |
| 10 x 10 MiB | 512 MiB / 5 | 1 | 1 | 1 |
| 5 x 128 MiB | 512 MiB / 5 | 1 | 1 | 1 |
MiB means 1024 * 1024 bytes. These results test selection/planning only; no Parquet files are read or rewritten.
### What doesn't meet your expectations?
Expected: compaction should be able to make progress on these contiguous normal files. Accumulating enough bytes for compaction should not repeatedly discard the group before it can satisfy the file-count condition.
Actual: every three files reach 600 MiB, exceeding the 512 MiB target. The group is flushed/drained, but rejected because it contains only 3 files (< 5). It is then cleared instead of carried forward. This repeats four times, so all 12 files produce zero candidates and zero tasks. Adding more files of the same size does not resolve this pattern.
This can make a compaction invocation finish without merging any of these files. There is no exception in the reproduction.
### Anything else?
Both selection stages contain this interaction:
1. [`CompactCandidateRangeCollector`](https://github.com/apache/paimon/blob/9c3c5187a4bd045d31b1ed9fb0decd58e6d63e8e/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/CompactCandidateRangeCollector.java#L540-L565) flushes the bin when its normal weight exceeds the target. [`CandidateBin.emit`](https://github.com/apache/paimon/blob/9c3c5187a4bd045d31b1ed9fb0decd58e6d63e8e/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/CompactCandidateRangeCollector.java#L635-L650) still requires the minimum normal-file count.
2. [`CompactPlanner`](https://github.com/apache/paimon/blob/9c3c5187a4bd045d31b1ed9fb0decd58e6d63e8e/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java#L373-L398) drains on the size threshold; [`triggerTask`](https://github.com/apache/paimon/blob/9c3c5187a4bd045d31b1ed9fb0decd58e6d63e8e/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java#L452-L466) then applies the file-count check.
Fixing only the final planner would leave the candidate prefilter blocking these files. One possible approach is to let a bin with multiple normal files trigger on accumulated size as well as file count, while retaining partition/row-ID boundaries and avoiding unnecessary single-file rewrites.
The large-file splitting support in #9709 does not cover this case: each 200 MiB file is smaller than the 512 MiB target. Enabling it with ratio 1 or 2 still produced zero candidates/tasks in the same harness.
This report concerns core Data Evolution file selection, separately from the compact_database routing issue #10102.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Data Evolution compaction skips contiguous small files due to size/count thresholds [paimon]
Posted by "hutiefang76 (via GitHub)" <gi...@apache.org>.
hutiefang76 commented on issue #10458:
URL: https://github.com/apache/paimon/issues/10458#issuecomment-6376498760
@tzphh, are you planning to continue #10459, or was it set aside for a specific reason? I saw it was closed as WIP while this issue remains open. I can check the current collector/planner against the 12 × 200 MiB case and a small Parquet compaction round trip, but want to avoid duplicating a newer branch or repeating an approach that was ruled out.
<sub>AI-generated comment.</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org