You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "liujiayi771 (via GitHub)" <gi...@apache.org> on 2026/08/09 08:54:59 UTC
[PR] [core][flink] Preserve file stats during primary-key compaction [paimon]
liujiayi771 opened a new pull request, #9131:
URL: https://github.com/apache/paimon/pull/9131
### Purpose
When `manifest.delete-file.drop-stats` is enabled, scan-level `dropStats()` strips stats from every scanned manifest entry, not just DELETE entries. During primary-key compaction, metadata-only level upgrades can reuse the same data file for DELETE and ADD entries. If compactor or restore scans drop the stats first, the ADD entry permanently loses file value stats and subsequent reads lose data-skipping metadata.
This change keeps stats on scan results and applies `copyWithoutStats()` only while constructing DELETE manifest entries. ADD entries therefore preserve file stats, while DELETE entries still omit them according to the configured option.
### Tests
- `ASDF_MAVEN_VERSION=3.9.9 mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=TableWriteTest#testDropStatsOnlyForDeleteManifestEntries test`
- `ASDF_MAVEN_VERSION=3.9.9 mvn -pl paimon-flink/paimon-flink-common -Pfast-build -Pflink1 -DwildcardSuites=none -Dtest=TableWriteCoordinatorTest#testScanPreservesStatsWhenDeleteManifestStatsAreDropped,CompactorSourceITCase#testStreamingReadPreservesStatsWhenDeleteManifestStatsAreDropped test`
- `git diff --check github/master...HEAD`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core][flink] Preserve file stats during primary-key compaction [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9131:
URL: https://github.com/apache/paimon/pull/9131#discussion_r3794135013
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/source/CompactorSourceBuilder.java:
##########
@@ -100,9 +100,6 @@ public CompactorSourceBuilder withPartitionIdleTime(@Nullable Duration partition
if (partitionPredicate != null) {
readBuilder.withPartitionFilter(partitionPredicate);
}
- if (CoreOptions.fromMap(table.options()).manifestDeleteFileDropStats()) {
- readBuilder = readBuilder.dropStats();
Review Comment:
[P2] Could we keep dropping stats for the bounded compactor? In batch mode, `CompactBucketsTable.BucketsRead` only checks the schema ID of each file and then emits an empty file list; `SIZE_AWARE_BATCH` uses only `fileSize`. Without `dropStats()`, bounded compaction retains full `valueStats` in the JobManager-planned `DataSplit`s and serializes them to source subtasks (and pending-split checkpoints, when enabled), only to discard them in the reader. With the default `metadata.stats-mode=truncate(16)` and `compaction.file-num-limit` allowing up to 100,000 files, this can materially increase JobManager heap, serialization CPU, and network usage for wide tables. A minimal fix would apply `readBuilder.dropStats()` only when `!isContinuous && manifestDeleteFileDropStats()`, plus add a batch-source regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core][flink] Preserve file stats during primary-key compaction [paimon]
Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on PR #9131:
URL: https://github.com/apache/paimon/pull/9131#issuecomment-5264256820
CC @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org