You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "talatuyarer (via GitHub)" <gi...@apache.org> on 2025/10/06 19:07:20 UTC
[PR] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
talatuyarer opened a new pull request, #14264:
URL: https://github.com/apache/iceberg/pull/14264
This PR extends `BaseIncrementalChangelogScan` to support positional and equality delete files (MoR use cases), enabling complete CDC coverage.
I read all discussion on @wypoon 's #10935 The PR was old and also it does not cover all cases. I decided to implement with a fresh approach but reduced scope. Initial PR had spark implementation. in this PR I will only focus on core module support. With this implementation, changelog scans now correctly produce `DeletedRowsScanTask`, `DeletedDataFileScanTask`, and `AddedRowsScanTask` even when delete files are present.
_I implemented unit tests for following scenario behaviors_
Scenario | Expected Behavior
-- | --
Insert → Equality Delete → Re-insert | One DELETE + one INSERT changelog event
Insert & Delete in Same Commit | Emitted as one task with attached deletes
Overlapping Equality + Position Deletes | Deduplicated delete application
Existing Deletes + New Deletes | Both applied without duplication
Overwrite Snapshot with Deletes | No NPE, correct ADDED/DELETED tasks
Large Delete Set | Pruned by partition filter
Step by Step Implementation Design:
1. Delete Index Construction
Two types of delete indexes are now built:
ExistingDeleteIndex – built from delete manifests before the scan start snapshot. Used to ensure previous deletes are applied but not re-emitted as changelog events. This helps keeping row lineage for CDC.
AddedDeletesBySnapshot – per-snapshot indexes of newly added delete files in each changelog snapshot.
2. Task Planning - Data File Changes
Each changelog snapshot incrementally accumulates delete files. `CreateDataFileChangeTasks` now attaches applicable deletes to each data file task:
- `AddedRowsScanTask` → combines existing + newly added deletes
- `DeletedDataFileScanTask` → includes existing deletes (before deletion)
3. Task Planning For Existing Data Files with New Deletes
planDeletedRowsTasks() identifies EXISTING data files affected by newly added delete files and emits corresponding DeletedRowsScanTask objects.
Processing multiple snapshot and equality deletes are expensive operations for huge tables. I added few performance optimization.
Optimization | Description
-- | --
Manifest caching | Introduced DeleteManifestCache to avoid redundant manifest parsing
Partition pruning | Uses filter pushdown to skip irrelevant delete manifests
Incremental accumulation | Avoids rebuilding the delete index for every snapshot
Selective copy | ContentFileUtil.copy() keeps only essential stats to reduce memory footprint
Possible Future Work
- V3 Deletion Vector support. I am trying to define foundation for Deletion Vector base CDC.
- Add persistent delete index cache
- Engine level support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2418337987
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
I added Manifest pruning for buildExistingDeleteIndex method fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2441197210
##########
core/src/main/java/org/apache/iceberg/DeleteManifestCache.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.TimeUnit;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+
+/**
+ * Cache for parsed delete file entries from manifests to avoid redundant manifest parsing across
+ * changelog scans.
+ *
+ * <p>This cache is thread-safe and uses LRU eviction with size and time-based expiration.
+ */
+class DeleteManifestCache {
Review Comment:
I removed the cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2441387283
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
It looks like there's another case where the existing index is necessary, which is where a file is deleted and we need to omit the records that were previously deleted (I forgot about that). I'm not sure if we can detect early in the same way we can for equality deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445437468
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
+ break;
+ }
+ }
+ if (needsExistingDeleteIndex) {
+ break;
+ }
+ }
+ }
+
+ if (!needsExistingDeleteIndex) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ } else {
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Process snapshots in order, tracking which files have been handled
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ // Apply partition pruning to only accumulate relevant delete files
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Build cumulative delete index for this snapshot from accumulated deletes
+ DeleteFileIndex cumulativeDeleteIndex = buildCumulativeDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ // Apply partition pruning to only accumulate relevant delete files
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a cumulative delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildCumulativeDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ String key = snapshot.snapshotId() + ":" + filePath;
+ if (alreadyProcessedPaths.contains(key)) {
+ continue;
+ }
+
+ // Check if this data file is affected by newly added delete files
+ DeleteFile[] addedDeletes = addedDeleteIndex.forEntry(entry);
+ if (addedDeletes.length == 0) {
+ continue;
+ }
+
+ // This data file was EXISTING but has new delete files applied
+ // Get existing deletes from before this snapshot (cumulative)
+ DeleteFile[] existingDeletes =
+ cumulativeDeleteIndex.isEmpty()
+ ? new DeleteFile[0]
+ : cumulativeDeleteIndex.forEntry(entry);
+
+ // Create a DeletedRowsScanTask
+ int changeOrdinal = snapshotOrdinals.get(snapshot.snapshotId());
+ String schemaString = SchemaParser.toJson(schema());
Review Comment:
We probably don't need to calculate this every time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-4791666524
Getting to this in next few days, still remember.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2417187369
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
I'm a little confused as to why we need to build the delete index before the start index. If a delete file exists prior to the start snapshot, it would only apply to data files that exist prior to selected scan range. This feels like it would never apply to the scan range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2438065787
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +155,468 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ if (fromSnapshot == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files with caching to avoid redundant manifest parsing
+ List<DeleteFile> deleteFiles = loadDeleteFilesWithCache(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ } else {
+ // Load delete files with caching to avoid redundant manifest parsing
+ List<DeleteFile> deleteFiles = loadDeleteFilesWithCache(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Process snapshots in order, tracking which files have been handled
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ // Apply partition pruning to only accumulate relevant delete files
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Build cumulative delete index for this snapshot from accumulated deletes
+ DeleteFileIndex cumulativeDeleteIndex = buildCumulativeDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ // Apply partition pruning to only accumulate relevant delete files
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a cumulative delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildCumulativeDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ String key = snapshot.snapshotId() + ":" + filePath;
+ if (alreadyProcessedPaths.contains(key)) {
+ continue;
+ }
+
+ // Check if this data file is affected by newly added delete files
+ DeleteFile[] addedDeletes = addedDeleteIndex.forEntry(entry);
+ if (addedDeletes.length == 0) {
+ continue;
+ }
+
+ // This data file was EXISTING but has new delete files applied
+ // Get existing deletes from before this snapshot (cumulative)
+ DeleteFile[] existingDeletes =
+ cumulativeDeleteIndex.isEmpty()
+ ? new DeleteFile[0]
+ : cumulativeDeleteIndex.forEntry(entry);
+
+ // Create a DeletedRowsScanTask
+ int changeOrdinal = snapshotOrdinals.get(snapshot.snapshotId());
+ String schemaString = SchemaParser.toJson(schema());
+ String specString = PartitionSpecParser.toJson(table().specs().get(dataFile.specId()));
+ PartitionSpec spec = table().specs().get(dataFile.specId());
+ Expression residualFilter = shouldIgnoreResiduals() ? Expressions.alwaysTrue() : filter();
+ ResidualEvaluator residuals = ResidualEvaluator.of(spec, residualFilter, isCaseSensitive());
+
+ tasks.add(
+ new BaseDeletedRowsScanTask(
+ changeOrdinal,
+ snapshot.snapshotId(),
+ dataFile.copy(shouldKeepStats()),
+ addedDeletes,
+ existingDeletes,
+ schemaString,
+ specString,
+ residuals));
+
+ // Mark this file+snapshot as processed
+ alreadyProcessedPaths.add(key);
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to plan deleted rows tasks", e);
+ }
+ }
+
+ private boolean shouldKeepStats() {
+ Set<Integer> columns = columnsToKeepStats();
+ return columns != null && !columns.isEmpty();
+ }
+
+ /**
+ * Loads delete files from manifests using a cache to avoid redundant manifest parsing. This
+ * significantly improves planning performance when the same manifests are accessed across
+ * multiple scans or within a scan range.
+ *
+ * @param manifests the delete manifests to load
+ * @return list of delete files
+ */
+ private List<DeleteFile> loadDeleteFilesWithCache(List<ManifestFile> manifests) {
+ DeleteManifestCache cache = DeleteManifestCache.instance();
+ List<DeleteFile> allDeleteFiles = Lists.newArrayList();
+
+ for (ManifestFile manifest : manifests) {
+ // Try to get from cache first
+ List<DeleteFile> cachedFiles = cache.get(manifest);
+
+ if (cachedFiles != null) {
+ // Cache hit - reuse parsed delete files
+ allDeleteFiles.addAll(cachedFiles);
+ } else {
+ // Cache miss - parse manifest and populate cache
+ List<DeleteFile> manifestDeleteFiles = loadDeleteFilesFromManifest(manifest);
+ cache.put(manifest, manifestDeleteFiles);
+ allDeleteFiles.addAll(manifestDeleteFiles);
+ }
+ }
+
+ return allDeleteFiles;
+ }
+
+ /**
+ * Prunes delete manifests based on partition filter to avoid processing irrelevant manifests.
+ * This significantly improves performance when only a subset of partitions are relevant to the
+ * scan.
+ *
+ * @param manifests all delete manifests to consider
+ * @return list of manifests that might contain relevant delete files
+ */
+ private List<ManifestFile> pruneManifestsByPartition(List<ManifestFile> manifests) {
+ Expression currentFilter = filter();
+
+ // If there's no filter, return all manifests
+ if (currentFilter == null || currentFilter.equals(Expressions.alwaysTrue())) {
+ return manifests;
+ }
+
+ List<ManifestFile> prunedManifests = Lists.newArrayList();
+
+ for (ManifestFile manifest : manifests) {
+ PartitionSpec spec = table().specs().get(manifest.partitionSpecId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // Include unpartitioned manifests
+ prunedManifests.add(manifest);
+ continue;
Review Comment:
I convered the flow to else if model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2441199846
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +156,464 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files with caching to avoid redundant manifest parsing
+ List<DeleteFile> deleteFiles = loadDeleteFilesWithCache(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
Review Comment:
nit: we do a lot of this pattern to represent an empty DeleteFileIndex. Should we just add a static "emptyIndex" (e.g. `DeleteFileIndex.emptyIndex()`) rather than rebuilding a new one every time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talip (via GitHub)" <gi...@apache.org>.
talip commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-3374464018
Changelog without deletes is useless! Thanks for adding this. Looking forward to it!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-4795224749
@talatuyarer @aokolnychyi will it be much easier if we only support v3 tables with deletion vector?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2483123777
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +82,14 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build per-snapshot delete file indexes for added deletes
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = buildAddedDeleteIndexes(changelogSnapshots);
+
+ // Build delete file index for existing deletes (before the start snapshot) if we have
+ // EQUALITY_DELETES files in the changelog range
+ DeleteFileIndex existingDeleteIndex =
+ buildExistingDeleteIndex(fromSnapshotIdExclusive, addedDeletesBySnapshot);
Review Comment:
@danielcweeks I've updated the unit tests and I believe I've covered all possible scenarios. Please let me know if you see any missing one.
@pvary The spec describes how position deletes work but does not forbid multiple deletes for the same position. Multiple position delete files can apply to the same data file, and When reading, all applicable position delete files are merged and positions are treated as a set - deleting the same position multiple times is idempotent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2446353511
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
Review Comment:
It feels like we're doing two passes over the deletes. Can't we integrate this into the first pass.
I also feel like this needs to be lazy (this isn't really lazy) since we may need to build this as we pass through the data files to determine if any of them have been removed.
I may be wrong, but we could potentially build this in at two different times:
1. building the existing delete index for the snapshots in the scan range if there any equality deletes found
2. building the delete index while processing the manifest files if there are any removed files
I feel like this should be possible just-in-time since a lot of CDF type use cases may apply to append only workloads and would avoid building the delete index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445439546
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
+ break;
+ }
+ }
+ if (needsExistingDeleteIndex) {
+ break;
+ }
+ }
+ }
+
+ if (!needsExistingDeleteIndex) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ } else {
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Process snapshots in order, tracking which files have been handled
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ // Apply partition pruning to only accumulate relevant delete files
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Build cumulative delete index for this snapshot from accumulated deletes
+ DeleteFileIndex cumulativeDeleteIndex = buildCumulativeDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ // Apply partition pruning to only accumulate relevant delete files
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a cumulative delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildCumulativeDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ String key = snapshot.snapshotId() + ":" + filePath;
+ if (alreadyProcessedPaths.contains(key)) {
+ continue;
+ }
+
+ // Check if this data file is affected by newly added delete files
+ DeleteFile[] addedDeletes = addedDeleteIndex.forEntry(entry);
+ if (addedDeletes.length == 0) {
+ continue;
+ }
+
+ // This data file was EXISTING but has new delete files applied
+ // Get existing deletes from before this snapshot (cumulative)
+ DeleteFile[] existingDeletes =
+ cumulativeDeleteIndex.isEmpty()
+ ? new DeleteFile[0]
+ : cumulativeDeleteIndex.forEntry(entry);
+
+ // Create a DeletedRowsScanTask
+ int changeOrdinal = snapshotOrdinals.get(snapshot.snapshotId());
+ String schemaString = SchemaParser.toJson(schema());
+ String specString = PartitionSpecParser.toJson(table().specs().get(dataFile.specId()));
Review Comment:
This is a bit more complicated, but maybe we don't need to calculate this again and again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445329999
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +156,464 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files with caching to avoid redundant manifest parsing
+ List<DeleteFile> deleteFiles = loadDeleteFilesWithCache(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2421096673
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
@pvary I believe I covered that use case with this unit test: `testMultipleSnapshotsWithDeletes`
I also have `testOverlappingEqualityAndPositionDeletes` too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "aarushigupta132 (via GitHub)" <gi...@apache.org>.
aarushigupta132 commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-5664866244
Also very keen on getting this feature! Thank you @talatuyarer for carrying the work, I would be happy to validate and help wherever needed to get this across
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-3477461998
@vigneshc Actually, this PR will provide full CDC support for V2 tables once it’s merged. We’ve already covered all possible scenarios. I haven’t implemented the engine execution part yet because I wanted to focus on the core functionality first. Would you consider contributing Spark support after this PR is merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2446784742
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +82,14 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build per-snapshot delete file indexes for added deletes
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = buildAddedDeleteIndexes(changelogSnapshots);
+
+ // Build delete file index for existing deletes (before the start snapshot) if we have
+ // EQUALITY_DELETES files in the changelog range
+ DeleteFileIndex existingDeleteIndex =
+ buildExistingDeleteIndex(fromSnapshotIdExclusive, addedDeletesBySnapshot);
Review Comment:
My main point was that we might need the existing deletes even when no equality delete is present.
> We shouldn't have overlapping positional deletes
@danielcweeks: Is this specifically forbidden by the spec, or it is just not reasonable, and it is probably not done by anyone? The only place where I can imagine this happening when we have concurrent deletes, and we decided that they can be committed without change as both of them are only deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445272107
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
Review Comment:
Instead of `break`, could we just return the value generated by a function?
This multi level breaks are really hard to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2535415803
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +185,512 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Wrapper method that tracks build calls and caches the result for reuse. This ensures we only
+ * build the index once even if called from multiple places.
+ */
+ private DeleteFileIndex buildExistingDeleteIndexTracked(Long fromSnapshotIdExclusive) {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ existingDeleteIndexBuildCallCount++;
+ cachedExistingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
+ return cachedExistingDeleteIndex;
+ }
+
+ // Visible for testing
+ int getExistingDeleteIndexBuildCallCount() {
+ return existingDeleteIndexBuildCallCount;
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount > 0;
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
Review Comment:
I don't think we should be using the guava streams where we can use the native streams implementations. Just use `snapshotDeleteManifests.stream().filter`
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +185,512 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Wrapper method that tracks build calls and caches the result for reuse. This ensures we only
+ * build the index once even if called from multiple places.
+ */
+ private DeleteFileIndex buildExistingDeleteIndexTracked(Long fromSnapshotIdExclusive) {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ existingDeleteIndexBuildCallCount++;
+ cachedExistingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
+ return cachedExistingDeleteIndex;
+ }
+
+ // Visible for testing
+ int getExistingDeleteIndexBuildCallCount() {
+ return existingDeleteIndexBuildCallCount;
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount > 0;
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
Review Comment:
Since we have to load the manifest lists for each snapshot, we should be doing this in parallel using `Tasks`
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -89,7 +121,32 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
manifestGroup = manifestGroup.planWith(planExecutor());
}
- return manifestGroup.plan(new CreateDataFileChangeTasks(changelogSnapshots));
+ // Create a supplier that reuses already-built index or builds lazily when first DELETED entry
+ // is encountered
+ Supplier<DeleteFileIndex> existingDeleteIndexSupplier =
+ () -> {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ return buildExistingDeleteIndexTracked(fromSnapshotIdExclusive);
+ };
+
+ // Plan data file tasks (ADDED and DELETED)
+ CloseableIterable<ChangelogScanTask> dataFileTasks =
+ manifestGroup.plan(
+ new CreateDataFileChangeTasks(
+ changelogSnapshots,
+ existingDeleteIndexSupplier,
+ addedDeletesBySnapshot,
+ table().specs(),
+ isCaseSensitive()));
+
+ // Find EXISTING data files affected by newly added delete files and create tasks for them
+ CloseableIterable<ChangelogScanTask> deletedRowsTasks =
+ planDeletedRowsTasks(
+ changelogSnapshots, existingDeleteIndex, addedDeletesBySnapshot, changelogSnapshotIds);
+
+ return CloseableIterable.concat(ImmutableList.of(dataFileTasks, deletedRowsTasks));
Review Comment:
It's a little unclear to me whether we're observing the correct orderedness by just concatenating these together. I'm not sure what the expectations are.
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -179,5 +735,56 @@ public CloseableIterable<ChangelogScanTask> apply(
}
});
}
+
+ /**
+ * Gets delete files that apply to an ADDED data file. Only includes deletes added in the same
+ * snapshot as the file.
+ */
+ private DeleteFile[] getDeletesForAddedFile(
+ ManifestEntry<DataFile> entry, long commitSnapshotId) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(commitSnapshotId);
+ return addedDeleteIndex == null || addedDeleteIndex.isEmpty()
+ ? NO_DELETES
+ : addedDeleteIndex.forEntry(entry);
+ }
+
+ /**
+ * Gets all delete files that were applied to a DELETED data file up to the point it was
+ * deleted. This includes existing deletes and all deletes added in the scan range up to (but
+ * not including) the deletion snapshot.
+ */
+ private DeleteFile[] getDeletesForDeletedFile(
+ ManifestEntry<DataFile> entry, long deletionSnapshotId) {
+
+ List<DeleteFile> allDeletes = Lists.newArrayList();
+
+ // Build existing delete index lazily when first DELETED entry is encountered
+ DeleteFileIndex existingDeleteIndex = existingDeleteIndexSupplier.get();
+ DeleteFile[] existingDeletes =
+ existingDeleteIndex.isEmpty() ? NO_DELETES : existingDeleteIndex.forEntry(entry);
+ for (DeleteFile df : existingDeletes) {
+ allDeletes.add(df);
+ }
+
+ // Add all deletes from snapshots in the scan range BEFORE the deletion
+ List<DeleteFile> cumulativeDeletes = cumulativeDeletesMap.get(deletionSnapshotId);
+ if (cumulativeDeletes != null) {
+ // For each cumulative delete, check if it applies to this entry
+ for (DeleteFile df : cumulativeDeletes) {
Review Comment:
This feels super expensive. Don't we know that it applies? If there's an existing POS/DV against a removed file, it would have to apply. If there's an equality delete, you would just include it regardless and let the task apply it?
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +185,512 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Wrapper method that tracks build calls and caches the result for reuse. This ensures we only
+ * build the index once even if called from multiple places.
+ */
+ private DeleteFileIndex buildExistingDeleteIndexTracked(Long fromSnapshotIdExclusive) {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ existingDeleteIndexBuildCallCount++;
+ cachedExistingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
+ return cachedExistingDeleteIndex;
+ }
+
+ // Visible for testing
+ int getExistingDeleteIndexBuildCallCount() {
+ return existingDeleteIndexBuildCallCount;
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount > 0;
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ } else {
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ DeleteFileIndex cumulativeDeleteIndex = buildDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ // Use a local set per snapshot to track processed files
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ String schemaString = SchemaParser.toJson(schema());
+
+ // Cache per specId - same for all files with same specId
+ Map<Integer, String> specStringCache = Maps.newHashMap();
+ Map<Integer, ResidualEvaluator> residualCache = Maps.newHashMap();
+ Expression residualFilter = shouldIgnoreResiduals() ? Expressions.alwaysTrue() : filter();
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ // Note: alreadyProcessedPaths is local to this snapshot's processing
+ if (alreadyProcessedPaths.contains(filePath)) {
+ continue;
+ }
+
+ // Check if this data file is affected by newly added delete files
+ DeleteFile[] addedDeletes = addedDeleteIndex.forEntry(entry);
+ if (addedDeletes.length == 0) {
+ continue;
+ }
+
+ // This data file was EXISTING but has new delete files applied
+ // Get existing deletes from before this snapshot (cumulative)
+ DeleteFile[] existingDeletes =
+ cumulativeDeleteIndex.isEmpty()
+ ? new DeleteFile[0]
+ : cumulativeDeleteIndex.forEntry(entry);
+
+ // Create a DeletedRowsScanTask
+ int changeOrdinal = snapshotOrdinals.get(snapshot.snapshotId());
+
+ // Use cached values (calculate once per specId)
+ int specId = dataFile.specId();
+ String specString =
+ specStringCache.computeIfAbsent(
+ specId, id -> PartitionSpecParser.toJson(table().specs().get(id)));
+ ResidualEvaluator residuals =
+ residualCache.computeIfAbsent(
+ specId,
+ id -> {
+ PartitionSpec spec = table().specs().get(id);
+ return ResidualEvaluator.of(spec, residualFilter, isCaseSensitive());
+ });
+
+ tasks.add(
+ new BaseDeletedRowsScanTask(
+ changeOrdinal,
+ snapshot.snapshotId(),
+ dataFile.copy(shouldKeepStats()),
+ addedDeletes,
+ existingDeletes,
+ schemaString,
+ specString,
+ residuals));
+
+ // Mark this file as processed for this snapshot
+ alreadyProcessedPaths.add(filePath);
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to plan deleted rows tasks", e);
+ }
+ }
+
+ private boolean shouldKeepStats() {
+ Set<Integer> columns = columnsToKeepStats();
+ return columns != null && !columns.isEmpty();
+ }
+
+ /**
+ * Loads delete files from manifests by parsing each manifest.
+ *
+ * @param manifests the delete manifests to load
+ * @return list of delete files
+ */
+ private List<DeleteFile> loadDeleteFiles(List<ManifestFile> manifests) {
Review Comment:
This feels like something we should be parallelizing using `Tasks`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "ericyangliu (via GitHub)" <gi...@apache.org>.
ericyangliu commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-4053112921
A much needed feature!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-4834236333
I went through the approach and it seems working. I had a slightly different mental model in my head where we would potentially do more upfront but simplify the rest of the logic. I am not sure it would be any better. Give me a few days to think about it and I will report back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-5649211018
@aokolnychyi I **reworked the planning** to follow the structure you describe.
The core of the change is a per-snapshot "effective delete list": `CumulativeDeleteIndexes` lazily builds, for each changelog snapshot, one index of base deletes plus deletes added earlier in the range, minus delete files removed earlier in the range. Both DeletedDataFileScanTask and DeletedRowsScanTask now resolve existing deletes from that shared index, which fixes the correctness issues you and @manuzhang identified (phantom removed deletes; missing base deletes when the range has no equality deletes).
Point-by-point vs your notes:
- `AddedRowsScanTask`: only same-snapshot deletes are attached; existing deletes are never scanned for new files.
- `DeletedDataFileScanTask`: existing deletes come from the effective delete list for the deletion snapshot, as you describe.
- `REWRITE outputs`: instead of reconstructing base + deltas, DeletedRows planning scans the snapshot's live data manifest list (partition-pruned), which includes mid-range compaction outputs by construction — so new deletes applying to compacted files are covered.
- Delete compactions mid-range: `REPLACE snapshots` are skipped, so their manifest churn never enters the accumulated state and the old, content-equivalent delete files are used.
I also implemented your pruning suggestions as well:
- The existing delete index is now scoped by the changes in the range: removed data files and added delete files define an affected partition set and an affected data file location set. Base delete manifests are pruned by the affected partitions on top of the user filter, and within the surviving manifests, file-scoped position deletes and DVs are loaded only if they reference an affected data file location (or lie in a partition where a partition-scoped delete was added), while equality and partition-scoped position deletes are loaded only for affected partitions.
- `DeletedRowsScanTask` planning prunes each snapshot's data manifest scan to the partitions of that snapshot's added delete files, and when every added delete is file-scoped (DVs or single-file position deletes), entries are further pruned to the referenced file locations — your "same partition set and file name filter" point.
- Net effect for DV-only workloads: the base delete scan touches only manifests and DVs for the files actually affected in the range, which is the outcome of your DV fast path. The scope is conservatively disabled when an added delete has global reach (an equality delete with an unpartitioned spec), and manifest-level partition pruning applies only while the affected partitions all belong to one spec.
Let me know if you see anything in the reworked structure that still diverges from your algorithm.
@manuzhang i also addressed your comments. Sorry for taking longer time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2441197785
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
I implemented lazy loading for existingDeleteIndex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445324956
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +82,14 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build per-snapshot delete file indexes for added deletes
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = buildAddedDeleteIndexes(changelogSnapshots);
+
+ // Build delete file index for existing deletes (before the start snapshot) if we have
+ // EQUALITY_DELETES files in the changelog range
+ DeleteFileIndex existingDeleteIndex =
+ buildExistingDeleteIndex(fromSnapshotIdExclusive, addedDeletesBySnapshot);
Review Comment:
We might need this when there is a positional delete in the existing delete index, and an equality delete later tries to delete the same row. In this case we don't emit the deleted row when the equality delete comes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2742880969
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +199,526 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Wrapper method that tracks build calls and caches the result for reuse. This ensures we only
+ * build the index once even if called from multiple places.
+ */
+ private DeleteFileIndex buildExistingDeleteIndexTracked(Long fromSnapshotIdExclusive) {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ existingDeleteIndexBuildCallCount++;
+ cachedExistingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
+ return cachedExistingDeleteIndex;
+ }
+
+ // Visible for testing
+ int getExistingDeleteIndexBuildCallCount() {
+ return existingDeleteIndexBuildCallCount;
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount > 0;
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newConcurrentMap();
+ Tasks.foreach(changelogSnapshots)
+ .retry(3)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .onFailure(
+ (snapshot, exc) ->
+ LOG.warn(
+ "Failed to build delete index for snapshot {}", snapshot.snapshotId(), exc))
+ .run(
+ snapshot -> {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ return;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ snapshotDeleteManifests.stream()
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .collect(Collectors.toUnmodifiableList());
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ } else {
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ });
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ DeleteFileIndex cumulativeDeleteIndex = buildDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ // Use a local set per snapshot to track processed files
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ String schemaString = SchemaParser.toJson(schema());
+
+ // Cache per specId - same for all files with same specId
+ Map<Integer, String> specStringCache = Maps.newHashMap();
+ Map<Integer, ResidualEvaluator> residualCache = Maps.newHashMap();
+ Expression residualFilter = shouldIgnoreResiduals() ? Expressions.alwaysTrue() : filter();
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ // Note: alreadyProcessedPaths is local to this snapshot's processing
+ if (alreadyProcessedPaths.contains(filePath)) {
+ continue;
+ }
+
+ // Check if this data file is affected by newly added delete files
+ DeleteFile[] addedDeletes = addedDeleteIndex.forEntry(entry);
+ if (addedDeletes.length == 0) {
+ continue;
+ }
+
+ // This data file was EXISTING but has new delete files applied
+ // Get existing deletes from before this snapshot (cumulative)
+ DeleteFile[] existingDeletes =
+ cumulativeDeleteIndex.isEmpty()
+ ? new DeleteFile[0]
+ : cumulativeDeleteIndex.forEntry(entry);
+
+ // Create a DeletedRowsScanTask
+ int changeOrdinal = snapshotOrdinals.get(snapshot.snapshotId());
+
+ // Use cached values (calculate once per specId)
+ int specId = dataFile.specId();
+ String specString =
+ specStringCache.computeIfAbsent(
+ specId, id -> PartitionSpecParser.toJson(table().specs().get(id)));
+ ResidualEvaluator residuals =
+ residualCache.computeIfAbsent(
+ specId,
+ id -> {
+ PartitionSpec spec = table().specs().get(id);
+ return ResidualEvaluator.of(spec, residualFilter, isCaseSensitive());
+ });
+
+ tasks.add(
+ new BaseDeletedRowsScanTask(
+ changeOrdinal,
+ snapshot.snapshotId(),
+ dataFile.copy(shouldKeepStats()),
+ addedDeletes,
+ existingDeletes,
+ schemaString,
+ specString,
+ residuals));
+
+ // Mark this file as processed for this snapshot
+ alreadyProcessedPaths.add(filePath);
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to plan deleted rows tasks", e);
+ }
+ }
+
+ private boolean shouldKeepStats() {
+ Set<Integer> columns = columnsToKeepStats();
+ return columns != null && !columns.isEmpty();
+ }
+
+ /**
+ * Loads delete files from manifests by parsing each manifest.
+ *
+ * @param manifests the delete manifests to load
+ * @return list of delete files
+ */
+ private Iterable<DeleteFile> loadDeleteFiles(List<ManifestFile> manifests) {
+ Queue<DeleteFile> allDeleteFiles = new ConcurrentLinkedQueue<>();
+
+ Tasks.foreach(manifests)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .run(
+ manifest -> {
+ List<DeleteFile> deleteFiles = loadDeleteFilesFromManifest(manifest);
+ allDeleteFiles.addAll(deleteFiles);
+ });
+
+ return allDeleteFiles;
+ }
+
+ /**
+ * Prunes delete manifests based on partition filter to avoid processing irrelevant manifests.
+ * This significantly improves performance when only a subset of partitions are relevant to the
+ * scan.
+ *
+ * @param manifests all delete manifests to consider
+ * @return list of manifests that might contain relevant delete files
+ */
+ private List<ManifestFile> pruneManifestsByPartition(List<ManifestFile> manifests) {
+ Expression currentFilter = filter();
+
+ // If there's no filter, return all manifests
+ if (currentFilter == null || currentFilter.equals(Expressions.alwaysTrue())) {
+ return manifests;
+ }
+
+ List<ManifestFile> prunedManifests = Lists.newArrayList();
+
+ for (ManifestFile manifest : manifests) {
+ PartitionSpec spec = table().specs().get(manifest.partitionSpecId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // Include unpartitioned manifests
+ prunedManifests.add(manifest);
+ } else if (manifestOverlapsFilter(manifest, spec, currentFilter)) {
+ // Check if manifest partition range overlaps with filter
+ prunedManifests.add(manifest);
+ }
+ }
+
+ return prunedManifests;
+ }
+
+ /**
+ * Checks if a manifest's partition range overlaps with the given filter.
+ *
+ * @param manifest the manifest to check
+ * @param spec the partition spec for the manifest
+ * @param filter the scan filter
+ * @return true if the manifest might contain matching partitions, false otherwise
+ */
+ private boolean manifestOverlapsFilter(
+ ManifestFile manifest, PartitionSpec spec, Expression filter) {
+ try {
+ // Use inclusive projection to transform row filter to partition filter
+ Expression partitionFilter = Projections.inclusive(spec, isCaseSensitive()).project(filter);
+
+ // Create evaluator for the partition filter
+ ManifestEvaluator evaluator =
+ ManifestEvaluator.forPartitionFilter(partitionFilter, spec, isCaseSensitive());
+
+ // Check if manifest could contain matching partitions
+ return evaluator.eval(manifest);
+ } catch (Exception e) {
+ // If evaluation fails, be conservative and include the manifest
+ return true;
+ }
+ }
+
+ /**
+ * Checks if a delete file's partition overlaps with the current scan filter. This enables
+ * partition pruning to reduce memory footprint and planning overhead by skipping delete files
+ * that cannot possibly match any rows in the scan.
+ *
+ * @param file the delete file to check
+ * @return true if the delete file's partition might contain matching rows, false otherwise
+ */
+ private boolean partitionMatchesFilter(DeleteFile file) {
+ // If there's no filter, all partitions match
+ Expression currentFilter = filter();
+ if (currentFilter == null || currentFilter.equals(Expressions.alwaysTrue())) {
+ return true;
+ }
+
+ // Get the partition spec for this delete file
+ PartitionSpec spec = table().specs().get(file.specId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // If spec not found or table is unpartitioned, be conservative and include the file
+ return true;
+ }
+
+ try {
+ // Project the row filter to partition space using inclusive projection
+ // This transforms expressions on source columns to expressions on partition columns
+ Expression partitionFilter =
+ Projections.inclusive(spec, isCaseSensitive()).project(currentFilter);
+
+ // Evaluate the projected filter against the delete file's partition
+ Evaluator evaluator = new Evaluator(spec.partitionType(), partitionFilter, isCaseSensitive());
+ return evaluator.eval(file.partition());
+ } catch (Exception e) {
+ // If evaluation fails, be conservative and include the file
+ return true;
+ }
+ }
+
+ /**
+ * Loads delete files from a single manifest, parsing the manifest entries.
+ *
+ * @param manifest the delete manifest to load
+ * @return list of delete files from this manifest
+ */
+ private List<DeleteFile> loadDeleteFilesFromManifest(ManifestFile manifest) {
+ List<DeleteFile> deleteFiles = Lists.newArrayList();
+
+ try (ManifestReader<DeleteFile> reader =
+ ManifestFiles.readDeleteManifest(manifest, table().io(), table().specs())) {
+ for (ManifestEntry<DeleteFile> entry : reader.entries()) {
+ if (entry.status() != ManifestEntry.Status.DELETED) {
+ // Only include live delete files, copy with minimal stats to save memory
+ DeleteFile file = entry.file();
+
+ // Apply partition pruning - skip delete files that cannot match the scan filter
+ if (!partitionMatchesFilter(file)) {
+ continue;
+ }
+
+ Set<Integer> columns =
+ file.content() == FileContent.POSITION_DELETES
+ ? Set.of(MetadataColumns.DELETE_FILE_PATH.fieldId())
+ : Set.copyOf(file.equalityFieldIds());
+ deleteFiles.add(ContentFileUtil.copy(file, true, columns));
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to read delete manifest: " + manifest.path(), e);
+ }
+
+ return deleteFiles;
+ }
+
private static class CreateDataFileChangeTasks implements CreateTasksFunction<ChangelogScanTask> {
private static final DeleteFile[] NO_DELETES = new DeleteFile[0];
private final Map<Long, Integer> snapshotOrdinals;
-
- CreateDataFileChangeTasks(Deque<Snapshot> snapshots) {
+ private final Supplier<DeleteFileIndex> existingDeleteIndexSupplier;
+ private final Map<Long, DeleteFileIndex> addedDeletesBySnapshot;
+ private final Map<Long, List<DeleteFile>> cumulativeDeletesMap;
Review Comment:
Say we have an operation that rewrites an equality delete file into a position delete file and replaces the deletes in the manifests. Given that we didn't track the equality delete file was removed, we may still assign it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2417382724
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
This is for equality delete case. The equality delete might created before the data file but applies to rows that exist in the new data file. I built previous delete file index to make sure all deletes applied for future files too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r3426929256
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +205,701 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
Review Comment:
existing delete index not built when there's no equality deletes such that pre-range position deletes will be missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r3426929256
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +205,701 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
Review Comment:
existing delete index not built when there's no equality deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r4078699823
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +270,851 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Manifests that cannot hold a delete relevant to the range are pruned before the index builder
+ * reads the rest, and files outside the affected scope are skipped as entries are read.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, ExistingDeleteScope scope) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
Review Comment:
This might have already been expired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2417576263
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
@pvary I'm not sure the case you provide is accurate because we would only be producing changes for S3, not snapshots prior to it. If this is incremental, it should only be the observed changes within the range, not changes prior to it. Since deletes only affect prior data, it would have no effect on the results of the scan.
You are correct that equality deletes do not apply to newer data, so even with equality deletes, it would only apply to older data that would not be part of the scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2417478771
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
>The equality delete might created before the data file but applies to rows that exist in the new data file.
This is not true. The equality delete does not apply for the new data files. The equality delete only applies for data files committed before the snapshot which adds the equality delete file.
**OTOH**
We still need to handle previous deletes. Consider the situation:
- Snapshot 1 (S1) adds Row 1 (R1)
- Snapshot 2 (S2) deletes R1
- Snapshot 3 (S3) adds an equality delete which would delete R1 - but does not, since it is already deleted
In this case the rows we should emit in a change log scan:
- S1 emits an insert for R1
- S2 emits a delete for R1
- S3 should not emit any row for R1
We can only calculate S3 if we know which rows were deleted before S3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2535724821
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -89,7 +121,32 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
manifestGroup = manifestGroup.planWith(planExecutor());
}
- return manifestGroup.plan(new CreateDataFileChangeTasks(changelogSnapshots));
+ // Create a supplier that reuses already-built index or builds lazily when first DELETED entry
+ // is encountered
+ Supplier<DeleteFileIndex> existingDeleteIndexSupplier =
+ () -> {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ return buildExistingDeleteIndexTracked(fromSnapshotIdExclusive);
+ };
+
+ // Plan data file tasks (ADDED and DELETED)
+ CloseableIterable<ChangelogScanTask> dataFileTasks =
+ manifestGroup.plan(
+ new CreateDataFileChangeTasks(
+ changelogSnapshots,
+ existingDeleteIndexSupplier,
+ addedDeletesBySnapshot,
+ table().specs(),
+ isCaseSensitive()));
+
+ // Find EXISTING data files affected by newly added delete files and create tasks for them
+ CloseableIterable<ChangelogScanTask> deletedRowsTasks =
+ planDeletedRowsTasks(
+ changelogSnapshots, existingDeleteIndex, addedDeletesBySnapshot, changelogSnapshotIds);
+
+ return CloseableIterable.concat(ImmutableList.of(dataFileTasks, deletedRowsTasks));
Review Comment:
It's a little unclear to me whether we're observing the correct orderedness by just concatenating these together. I'm not sure what the expectations are. When we calculate the snapshot range, it's explicit that we preserve order, but here we don't maintain that guarantee.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2438064764
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +155,468 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ if (fromSnapshot == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
Review Comment:
I addressed this my adding precondition check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2440631293
##########
core/src/main/java/org/apache/iceberg/DeleteManifestCache.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.TimeUnit;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+
+/**
+ * Cache for parsed delete file entries from manifests to avoid redundant manifest parsing across
+ * changelog scans.
+ *
+ * <p>This cache is thread-safe and uses LRU eviction with size and time-based expiration.
+ */
+class DeleteManifestCache {
Review Comment:
I agreed with you. 1K is too small for real production use case. I hesitate to put memory pressure on worker. I added caching before implementing partition pruning for delete indexes. Do you want me to make cache size configurable by user or Remove the cache entirely ?
IMO changelog scans are typically few snapshots except initial starts if users use streaming style for CDC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445354086
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
+ break;
+ }
+ }
+ if (needsExistingDeleteIndex) {
+ break;
+ }
+ }
+ }
+
+ if (!needsExistingDeleteIndex) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ } else {
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Process snapshots in order, tracking which files have been handled
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ // Apply partition pruning to only accumulate relevant delete files
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Build cumulative delete index for this snapshot from accumulated deletes
+ DeleteFileIndex cumulativeDeleteIndex = buildCumulativeDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ // Apply partition pruning to only accumulate relevant delete files
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
Review Comment:
Should we already apply the partition filter for the delete index?
If we do so, we don't have to do it again and again, and we can we can handle smaller amount of files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r3426991220
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -89,7 +132,41 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
manifestGroup = manifestGroup.planWith(planExecutor());
}
- return manifestGroup.plan(new CreateDataFileChangeTasks(changelogSnapshots));
+ // Create a supplier that reuses already-built index or builds lazily when first DELETED entry
+ // is encountered
+ Supplier<DeleteFileIndex> existingDeleteIndexSupplier =
+ () -> {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ return buildExistingDeleteIndexTracked(fromSnapshotIdExclusive);
+ };
+
+ // Plan data file tasks (ADDED and DELETED)
+ Map<Long, List<DeleteFile>> cumulativeDeletesMap =
+ buildCumulativeDeletesBySnapshot(changelogSnapshots, addedDeletesBySnapshot);
+
+ CloseableIterable<ChangelogScanTask> dataFileTasks =
+ manifestGroup.plan(
+ new CreateDataFileChangeTasks(
+ changelogSnapshots,
+ existingDeleteIndexSupplier,
+ addedDeletesBySnapshot,
+ cumulativeDeletesMap,
+ table().specs(),
+ isCaseSensitive()));
+
+ // Find EXISTING data files affected by newly added delete files and create tasks for them
+ CloseableIterable<ChangelogScanTask> deletedRowsTasks =
+ planDeletedRowsTasks(
+ changelogSnapshots, existingDeleteIndex, addedDeletesBySnapshot, changelogSnapshotIds);
+
+ // Merge tasks from both iterables in order by changeOrdinal
+ Comparator<ChangelogScanTask> byOrdinal =
+ Comparator.comparing(ChangelogScanTask::changeOrdinal)
+ .thenComparing(ChangelogScanTask::commitSnapshotId);
+
+ return new SortedMerge<>(byOrdinal, ImmutableList.of(dataFileTasks, deletedRowsTasks));
Review Comment:
This assumes the passed-in iterables are in order but dataFileTasks are not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-5248279491
@talatuyarer I think there are some unresolved comments that needs to be addressed first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2535808483
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -179,5 +735,56 @@ public CloseableIterable<ChangelogScanTask> apply(
}
});
}
+
+ /**
+ * Gets delete files that apply to an ADDED data file. Only includes deletes added in the same
+ * snapshot as the file.
+ */
+ private DeleteFile[] getDeletesForAddedFile(
+ ManifestEntry<DataFile> entry, long commitSnapshotId) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(commitSnapshotId);
+ return addedDeleteIndex == null || addedDeleteIndex.isEmpty()
+ ? NO_DELETES
+ : addedDeleteIndex.forEntry(entry);
+ }
+
+ /**
+ * Gets all delete files that were applied to a DELETED data file up to the point it was
+ * deleted. This includes existing deletes and all deletes added in the scan range up to (but
+ * not including) the deletion snapshot.
+ */
+ private DeleteFile[] getDeletesForDeletedFile(
+ ManifestEntry<DataFile> entry, long deletionSnapshotId) {
+
+ List<DeleteFile> allDeletes = Lists.newArrayList();
+
+ // Build existing delete index lazily when first DELETED entry is encountered
+ DeleteFileIndex existingDeleteIndex = existingDeleteIndexSupplier.get();
+ DeleteFile[] existingDeletes =
+ existingDeleteIndex.isEmpty() ? NO_DELETES : existingDeleteIndex.forEntry(entry);
+ for (DeleteFile df : existingDeletes) {
+ allDeletes.add(df);
+ }
+
+ // Add all deletes from snapshots in the scan range BEFORE the deletion
+ List<DeleteFile> cumulativeDeletes = cumulativeDeletesMap.get(deletionSnapshotId);
+ if (cumulativeDeletes != null) {
+ // For each cumulative delete, check if it applies to this entry
+ for (DeleteFile df : cumulativeDeletes) {
Review Comment:
This feels super expensive. Don't we know that it applies? If there's an existing POS/DV against a removed file, it would have to apply. If there's an equality delete, you would just include it regardless and let the task apply it? Overall, it feels like any associated delete would apply or could/would be deconflicted by the task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445370306
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
+ break;
+ }
+ }
+ if (needsExistingDeleteIndex) {
+ break;
+ }
+ }
+ }
+
+ if (!needsExistingDeleteIndex) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ } else {
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Process snapshots in order, tracking which files have been handled
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ // Apply partition pruning to only accumulate relevant delete files
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Build cumulative delete index for this snapshot from accumulated deletes
+ DeleteFileIndex cumulativeDeleteIndex = buildCumulativeDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ // Apply partition pruning to only accumulate relevant delete files
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a cumulative delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildCumulativeDeleteIndex(List<DeleteFile> accumulatedDeletes) {
Review Comment:
nit: This is a simple DeleteFileIndex. Nothing to do with how the input is calculated.
Maybe:
`buildDeleteIndex`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2483123910
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
+ break;
+ }
+ }
+ if (needsExistingDeleteIndex) {
+ break;
+ }
+ }
+ }
+
+ if (!needsExistingDeleteIndex) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ } else {
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Process snapshots in order, tracking which files have been handled
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ // Apply partition pruning to only accumulate relevant delete files
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Build cumulative delete index for this snapshot from accumulated deletes
+ DeleteFileIndex cumulativeDeleteIndex = buildCumulativeDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ // Apply partition pruning to only accumulate relevant delete files
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a cumulative delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildCumulativeDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ String key = snapshot.snapshotId() + ":" + filePath;
+ if (alreadyProcessedPaths.contains(key)) {
+ continue;
+ }
+
+ // Check if this data file is affected by newly added delete files
+ DeleteFile[] addedDeletes = addedDeleteIndex.forEntry(entry);
+ if (addedDeletes.length == 0) {
+ continue;
+ }
+
+ // This data file was EXISTING but has new delete files applied
+ // Get existing deletes from before this snapshot (cumulative)
+ DeleteFile[] existingDeletes =
+ cumulativeDeleteIndex.isEmpty()
+ ? new DeleteFile[0]
+ : cumulativeDeleteIndex.forEntry(entry);
+
+ // Create a DeletedRowsScanTask
+ int changeOrdinal = snapshotOrdinals.get(snapshot.snapshotId());
+ String schemaString = SchemaParser.toJson(schema());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2483124127
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
+ break;
+ }
+ }
+ if (needsExistingDeleteIndex) {
+ break;
+ }
+ }
+ }
+
+ if (!needsExistingDeleteIndex) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ } else {
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Process snapshots in order, tracking which files have been handled
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ // Apply partition pruning to only accumulate relevant delete files
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Build cumulative delete index for this snapshot from accumulated deletes
+ DeleteFileIndex cumulativeDeleteIndex = buildCumulativeDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ // Apply partition pruning to only accumulate relevant delete files
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a cumulative delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildCumulativeDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ String key = snapshot.snapshotId() + ":" + filePath;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "vigneshc (via GitHub)" <gi...@apache.org>.
vigneshc commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-3476771461
Sightly unrelated to the PR.
I am trying to understand the state and roadmap of changelog / incremental cdc support in core and spark.
Looking at this PR's description, it looks like there is still quite a bit to do.
What is a good resource for me to understand it? I am looking for both documentation level content and design details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-3821104935
What I like about this PR is the attempt to have per snapshot state like `DeleteFileIndex` and the incremental approach for building that state. It does seem like there are some correctness issues in the implementation, however.
I found some old notes on this topic and spent some time thinking about it. Sharing them below.
## CDC tasks
- `AddedRowsScanTask` (data file + deletes assigned in the same snapshot)
- `DeletedDataFileScanTask` (data file + existing deletes)
- `DeletedRowsScanTask` (data file + added deletes + existing deletes)
## Potential Algorithm
Iterate over changelog snapshots (ignoring logical rewrites).
- Find new data and delete files added across the changelog range (look for new manifests and ADDED manifest entry statuses).
- Find data and delete files that have been removed across the changelog range (look for new manifests and DELETED manifest entry statuses).
- Build the affected partition set and a set of affected data files locations for pruning of base data and delete manifests if there are removed data files or new deletes. This combined predicate may be used for pruning along the user-provided filter. It would essentially tell us what partitions have been affected.
### AddedRowsScanTask generation
The only delete lookup that needs to happen for new data files is DVs associated in the same snapshot. Like in this PR, we can simply build `DeleteFileIndex` using the new delete files from that snapshot. This information will be sufficient to emit this type of tasks. We don’t need to analyze existing deletes here.
### DeletedDataFileScanTask generation
For each removed data file, we need to find deletes that were associated with it. We have to scan the effective delete file list for this changelog snapshot (base + incremental changes up to this snapshot) while doing the lookup. We don’t have manifest affinity in V3 but we can prune the base delete set using the user provided predicate and by collecting partitions and file names of all affected data files in the changelog range.
If the table only contains DVs, the set of removed delete files for that snapshot will contain the old DV. While dropping data files engines must drop the DV too (double check this statement). This means we don’t have to scan through base deletes if we know that we only deal with DVs.
It is likely OK to ignore delete compactions that happened during the changelog range and use old deletes as their content should be the same.
### DeletedRowsScanTask generation
For each new delete file, we have to find its data file and the list of previous deletes.
If the added delete is a DV, the old DV (if existed) must be in the set of removed DVs in that snapshot. We won’t have to scan historic deletes.
We can use the same partition set and file name filter to find data files. We also have to scan the effective set of data files. The applicable data file will have to be either in the base file set or in the ANY snapshot after that including REWRITEs that we ignore while selecting changelog snapshots. Therefore, we will have to check compaction outputs, newly added files in the changelog range and potentially the base set of data files (filtered by affected files and partitions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2418219303
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
@pvary and @danielcweeks We need previous Delete indexes for `Overlapping Equality + Position Deletes` scenario. Let says I deleted a row with positional delete file (lets says there is a id column i delete id = 5) before my scan range. In my scan range i delete all rows with equality delete ( lets assume I deleted id > 1) We should not emit DELETE two times for id =5 row because it is already deleted before my scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2417187369
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
I'm a little confused as to why we need to build the delete index before the start snapshot. If a delete file exists prior to the start snapshot, it would only apply to data files that exist prior to selected scan range. This feels like it would never apply to the scan range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2437513072
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +155,468 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ if (fromSnapshot == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files with caching to avoid redundant manifest parsing
+ List<DeleteFile> deleteFiles = loadDeleteFilesWithCache(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ } else {
+ // Load delete files with caching to avoid redundant manifest parsing
+ List<DeleteFile> deleteFiles = loadDeleteFilesWithCache(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Process snapshots in order, tracking which files have been handled
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ // Apply partition pruning to only accumulate relevant delete files
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Build cumulative delete index for this snapshot from accumulated deletes
+ DeleteFileIndex cumulativeDeleteIndex = buildCumulativeDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ // Apply partition pruning to only accumulate relevant delete files
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a cumulative delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildCumulativeDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ String key = snapshot.snapshotId() + ":" + filePath;
+ if (alreadyProcessedPaths.contains(key)) {
+ continue;
+ }
+
+ // Check if this data file is affected by newly added delete files
+ DeleteFile[] addedDeletes = addedDeleteIndex.forEntry(entry);
+ if (addedDeletes.length == 0) {
+ continue;
+ }
+
+ // This data file was EXISTING but has new delete files applied
+ // Get existing deletes from before this snapshot (cumulative)
+ DeleteFile[] existingDeletes =
+ cumulativeDeleteIndex.isEmpty()
+ ? new DeleteFile[0]
+ : cumulativeDeleteIndex.forEntry(entry);
+
+ // Create a DeletedRowsScanTask
+ int changeOrdinal = snapshotOrdinals.get(snapshot.snapshotId());
+ String schemaString = SchemaParser.toJson(schema());
+ String specString = PartitionSpecParser.toJson(table().specs().get(dataFile.specId()));
+ PartitionSpec spec = table().specs().get(dataFile.specId());
+ Expression residualFilter = shouldIgnoreResiduals() ? Expressions.alwaysTrue() : filter();
+ ResidualEvaluator residuals = ResidualEvaluator.of(spec, residualFilter, isCaseSensitive());
+
+ tasks.add(
+ new BaseDeletedRowsScanTask(
+ changeOrdinal,
+ snapshot.snapshotId(),
+ dataFile.copy(shouldKeepStats()),
+ addedDeletes,
+ existingDeletes,
+ schemaString,
+ specString,
+ residuals));
+
+ // Mark this file+snapshot as processed
+ alreadyProcessedPaths.add(key);
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to plan deleted rows tasks", e);
+ }
+ }
+
+ private boolean shouldKeepStats() {
+ Set<Integer> columns = columnsToKeepStats();
+ return columns != null && !columns.isEmpty();
+ }
+
+ /**
+ * Loads delete files from manifests using a cache to avoid redundant manifest parsing. This
+ * significantly improves planning performance when the same manifests are accessed across
+ * multiple scans or within a scan range.
+ *
+ * @param manifests the delete manifests to load
+ * @return list of delete files
+ */
+ private List<DeleteFile> loadDeleteFilesWithCache(List<ManifestFile> manifests) {
+ DeleteManifestCache cache = DeleteManifestCache.instance();
+ List<DeleteFile> allDeleteFiles = Lists.newArrayList();
+
+ for (ManifestFile manifest : manifests) {
+ // Try to get from cache first
+ List<DeleteFile> cachedFiles = cache.get(manifest);
+
+ if (cachedFiles != null) {
+ // Cache hit - reuse parsed delete files
+ allDeleteFiles.addAll(cachedFiles);
+ } else {
+ // Cache miss - parse manifest and populate cache
+ List<DeleteFile> manifestDeleteFiles = loadDeleteFilesFromManifest(manifest);
+ cache.put(manifest, manifestDeleteFiles);
+ allDeleteFiles.addAll(manifestDeleteFiles);
+ }
+ }
+
+ return allDeleteFiles;
+ }
+
+ /**
+ * Prunes delete manifests based on partition filter to avoid processing irrelevant manifests.
+ * This significantly improves performance when only a subset of partitions are relevant to the
+ * scan.
+ *
+ * @param manifests all delete manifests to consider
+ * @return list of manifests that might contain relevant delete files
+ */
+ private List<ManifestFile> pruneManifestsByPartition(List<ManifestFile> manifests) {
+ Expression currentFilter = filter();
+
+ // If there's no filter, return all manifests
+ if (currentFilter == null || currentFilter.equals(Expressions.alwaysTrue())) {
+ return manifests;
+ }
+
+ List<ManifestFile> prunedManifests = Lists.newArrayList();
+
+ for (ManifestFile manifest : manifests) {
+ PartitionSpec spec = table().specs().get(manifest.partitionSpecId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // Include unpartitioned manifests
+ prunedManifests.add(manifest);
+ continue;
Review Comment:
nit: this control-flow feels awkward and should probably be and `else if`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "vigneshc (via GitHub)" <gi...@apache.org>.
vigneshc commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-3492091046
> @vigneshc Actually, this PR will provide full CDC support for V2 tables once it’s merged. We’ve already covered all possible scenarios. I haven’t implemented the engine execution part yet because I wanted to focus on the core functionality first. Would you consider contributing Spark support after this PR is merged?
I see this procedure, create_changelog_view . https://iceberg.apache.org/docs/latest/spark-procedures/#change-data-capture
Can you help me understand the difference between the support this procedure offers, and the pr enables? Initial pointers are enough.
On spark, I have not contributed to this repo, not sure how hard it would be. I can give it a shot with some help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2418312507
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
I thought about `existingDeleteIndex`. Currently All delete manifests are loaded and parsed upfront, regardless of whether they're relevant to the scan filter. For tables with thousands of delete manifests, this can be extremely slow. Even we have caching, the first scan still parses all manifests. Memory footprint can be also big for huge tables.
We can optimize planning performance by adding manifest with partition filters. However there could be still big memory need if there is so many delete fields for specific partitions.
I can also implement lazy loading for deletefiles that loads delete files only when needed during planning.
Do you think we should reduce scope for this PR and put basic functionality for CDC and I can create more PR for performance improvements ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2483124044
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
+ break;
+ }
+ }
+ if (needsExistingDeleteIndex) {
+ break;
+ }
+ }
+ }
+
+ if (!needsExistingDeleteIndex) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ } else {
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Process snapshots in order, tracking which files have been handled
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ // Apply partition pruning to only accumulate relevant delete files
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Build cumulative delete index for this snapshot from accumulated deletes
+ DeleteFileIndex cumulativeDeleteIndex = buildCumulativeDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ // Apply partition pruning to only accumulate relevant delete files
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a cumulative delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildCumulativeDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ String key = snapshot.snapshotId() + ":" + filePath;
+ if (alreadyProcessedPaths.contains(key)) {
+ continue;
+ }
+
+ // Check if this data file is affected by newly added delete files
+ DeleteFile[] addedDeletes = addedDeleteIndex.forEntry(entry);
+ if (addedDeletes.length == 0) {
+ continue;
+ }
+
+ // This data file was EXISTING but has new delete files applied
+ // Get existing deletes from before this snapshot (cumulative)
+ DeleteFile[] existingDeletes =
+ cumulativeDeleteIndex.isEmpty()
+ ? new DeleteFile[0]
+ : cumulativeDeleteIndex.forEntry(entry);
+
+ // Create a DeletedRowsScanTask
+ int changeOrdinal = snapshotOrdinals.get(snapshot.snapshotId());
+ String schemaString = SchemaParser.toJson(schema());
+ String specString = PartitionSpecParser.toJson(table().specs().get(dataFile.specId()));
Review Comment:
I also cache this line now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2421096673
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
@pvary I believe I covered that use case with this unit test: `testMultipleSnapshotsWithDeletes`
I also have testOverlappingEqualityAndPositionDeletes too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-3373466011
@wypoon @pvary @aokolnychyi @stevenzwu I will appreciated if you share your thoughts. I believe This PR will unblock many use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445313971
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
+ break;
+ }
+ }
+ if (needsExistingDeleteIndex) {
+ break;
+ }
+ }
+ }
+
+ if (!needsExistingDeleteIndex) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
Review Comment:
Maybe it is theoretical, but could we have a situation where we had an existing delete file which is removed in the latest snapshot?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445313971
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
+ break;
+ }
+ }
+ if (needsExistingDeleteIndex) {
+ break;
+ }
+ }
+ }
+
+ if (!needsExistingDeleteIndex) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
Review Comment:
Maybe it is theoretical, but could we have a situation where we had an existing delete file which is removed in the latest snapshot?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2446335553
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +82,14 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build per-snapshot delete file indexes for added deletes
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = buildAddedDeleteIndexes(changelogSnapshots);
+
+ // Build delete file index for existing deletes (before the start snapshot) if we have
+ // EQUALITY_DELETES files in the changelog range
+ DeleteFileIndex existingDeleteIndex =
+ buildExistingDeleteIndex(fromSnapshotIdExclusive, addedDeletesBySnapshot);
Review Comment:
@pvary That's correct, but I think what we're trying to achieve is only build the existing deletion index when required (not all the time). We shouldn't have overlapping positional deletes, but an equality delete can overlap (either an existing positional delete or a previous equality delete) where we should not emit a row.
We also need to handle the case where there was a whole file deleted, which also had some number of deletes applied to it (positional or equality) so we omit the deleted rows.
@talatuyarer we need to add test cases for these scenarios, which we appear to not properly cover since our tests are passing, but we're not covering all these cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r4078681993
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +270,851 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Manifests that cannot hold a delete relevant to the range are pruned before the index builder
+ * reads the rest, and files outside the affected scope are skipped as entries are read.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, ExistingDeleteScope scope) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> deleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (deleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests that cannot contain deletes for any partition affected by the scan range
+ if (scope != null) {
+ deleteManifests =
+ pruneManifestsByAffectedPartitions(deleteManifests, scope.allAffectedPartitions);
+ }
+
+ if (deleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // filterData prunes manifests and entries against the scan filter with per-spec cached
+ // evaluators and keeps minimal stats. Entry pruning also runs the filter against each delete
+ // file's own stats, which may drop a delete file whose bounds cannot match the filter; that is
+ // only safe because the task residual re-applies the filter to the rows a task emits, so
+ // ignoreResiduals must be forwarded (as ManifestGroup does) to disable it when the residual is
+ // dropped. Deliberately no planWith: this may run lazily under the cumulative-index monitor on
+ // shared worker-pool threads, and nested submission to the same pool can starve it
+ DeleteFileIndex.Builder builder =
+ DeleteFileIndex.builderFor(table().io(), deleteManifests)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .filterData(filter());
+
+ if (shouldIgnoreResiduals()) {
+ builder.ignoreResiduals();
+ }
+
+ if (scope != null) {
+ // skip delete files that cannot apply to any task in the scan range
+ builder.deleteFilePredicate(scope::keeps);
+ }
+
+ return builder.build();
+ }
+
+ /**
+ * Builds the existing (pre-range) delete index at most once per planFiles() invocation, so that
+ * concurrent planning threads share one index while a later plan of the same scan, which may
+ * resolve a different range and scope, builds its own.
+ */
+ private class ExistingDeleteIndexHolder {
+ private final Long fromSnapshotIdExclusive;
+ private final ExistingDeleteScope scope;
+ private DeleteFileIndex index = null;
+
+ ExistingDeleteIndexHolder(Long fromSnapshotIdExclusive, ExistingDeleteScope scope) {
+ this.fromSnapshotIdExclusive = fromSnapshotIdExclusive;
+ this.scope = scope;
+ }
+
+ synchronized DeleteFileIndex get() {
+ if (index == null) {
+ this.index = buildExistingDeleteIndex(fromSnapshotIdExclusive, scope);
+ existingDeleteIndexBuildCallCount.incrementAndGet();
+ }
+
+ return index;
+ }
+ }
+
+ // Visible for testing
+ int existingDeleteIndexBuildCount() {
+ return existingDeleteIndexBuildCallCount.get();
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount.get() > 0;
+ }
+
+ /** Added delete indexes and removed delete files, per snapshot, read in one manifest pass. */
+ private static class DeleteFileChanges {
+ private final Map<Long, DeleteFileIndex> addedBySnapshot;
+ private final Map<Long, List<DeleteFile>> removedBySnapshot;
+
+ DeleteFileChanges(
+ Map<Long, DeleteFileIndex> addedBySnapshot, Map<Long, List<DeleteFile>> removedBySnapshot) {
+ this.addedBySnapshot = addedBySnapshot;
+ this.removedBySnapshot = removedBySnapshot;
+ }
+ }
+
+ /**
+ * Reads the delete manifests written by each changelog snapshot exactly once, splitting live
+ * entries into the snapshot's added delete files (indexed) and removed delete files.
+ */
+ private DeleteFileChanges collectDeleteFileChanges(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedBySnapshot = Maps.newConcurrentMap();
+ Map<Long, List<DeleteFile>> removedBySnapshot = Maps.newConcurrentMap();
+
+ // one evaluator of the scan filter per partition spec, shared by all manifests and snapshots
+ Map<Integer, Evaluator> partitionEvaluators = Maps.newConcurrentMap();
+
+ Tasks.foreach(changelogSnapshots)
+ .retry(3)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .onFailure(
+ (snapshot, exc) ->
+ LOG.warn(
+ "Failed to read delete manifests for snapshot {}", snapshot.snapshotId(), exc))
+ .run(
+ snapshot -> {
+ long snapshotId = snapshot.snapshotId();
+ List<ManifestFile> newDeleteManifests =
+ snapshot.deleteManifests(table().io()).stream()
+ .filter(
+ manifest ->
+ manifest.snapshotId() != null && manifest.snapshotId() == snapshotId)
+ .toList();
+
+ List<DeleteFile> added = Lists.newArrayList();
+ List<DeleteFile> removed = Lists.newArrayList();
+ for (ManifestFile manifest : newDeleteManifests) {
+ readDeleteManifest(manifest, snapshotId, partitionEvaluators, added, removed);
+ }
+
+ addedBySnapshot.put(
+ snapshotId,
+ added.isEmpty()
+ ? DeleteFileIndex.emptyIndex()
+ : DeleteFileIndex.builderFor(added)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build());
+ removedBySnapshot.put(snapshotId, removed);
+ });
+
+ return new DeleteFileChanges(addedBySnapshot, removedBySnapshot);
+ }
+
+ /**
+ * Reads one delete manifest, collecting the delete files this snapshot added and removed. Only
+ * entries committed by the given snapshot are considered; EXISTING entries carried over from
+ * earlier snapshots keep their original snapshot ID and are skipped.
+ */
+ private void readDeleteManifest(
+ ManifestFile manifest,
+ long snapshotId,
+ Map<Integer, Evaluator> partitionEvaluators,
+ List<DeleteFile> added,
+ List<DeleteFile> removed) {
+ try (ManifestReader<DeleteFile> reader =
+ ManifestFiles.readDeleteManifest(manifest, table().io(), table().specs())) {
+ for (ManifestEntry<DeleteFile> entry : reader.entries()) {
+ if (entry.snapshotId() == null || entry.snapshotId() != snapshotId) {
+ continue;
+ }
+
+ DeleteFile file = entry.file();
+
+ // Apply partition pruning - skip delete files that cannot match the scan filter
+ if (!partitionMatchesFilter(file, partitionEvaluators)) {
+ continue;
+ }
+
+ Set<Integer> columns =
+ file.content() == FileContent.POSITION_DELETES
+ ? Set.of(MetadataColumns.DELETE_FILE_PATH.fieldId())
+ : Set.copyOf(file.equalityFieldIds());
+ DeleteFile copied = ContentFileUtil.copy(file, true, columns);
+
+ if (entry.status() == ManifestEntry.Status.DELETED) {
+ removed.add(copied);
+ } else {
+ added.add(copied);
+ }
+ }
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to read delete manifest: " + manifest.path(), e);
+ }
+ }
+
+ /**
+ * Returns true if the delete file's partition may contain rows matching the scan filter, so that
+ * delete files that cannot apply to any scanned row are skipped. The scan filter is projected
+ * into each spec's partition space once and the resulting evaluator is cached in the given map,
+ * which may be shared by concurrent readers.
+ */
+ private boolean partitionMatchesFilter(DeleteFile file, Map<Integer, Evaluator> evaluatorCache) {
+ Expression currentFilter = filter();
+ if (currentFilter.equals(Expressions.alwaysTrue())) {
+ return true;
+ }
+
+ PartitionSpec spec = table().specs().get(file.specId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // if the spec is unknown or the table is unpartitioned, be conservative and keep the file
+ return true;
+ }
+
+ try {
+ Evaluator evaluator =
+ evaluatorCache.computeIfAbsent(
+ file.specId(),
+ ignored -> {
+ // an inclusive projection turns predicates on source columns into predicates on
+ // partition values, so non-identity transforms are handled correctly
+ Expression partitionFilter =
+ Projections.inclusive(spec, isCaseSensitive()).project(currentFilter);
+ return new Evaluator(spec.partitionType(), partitionFilter, isCaseSensitive());
+ });
+ return evaluator.eval(file.partition());
+ } catch (Exception e) {
+ // if the filter cannot be projected or evaluated, be conservative and keep the file
+ return true;
+ }
+ }
+
+ /**
+ * Returns the partitions of the added delete files, or an empty set when some delete has global
+ * reach (an unpartitioned spec) and data manifests cannot be pruned by partition.
+ */
+ private PartitionSet addedDeletePartitions(DeleteFileIndex addedDeleteIndex) {
+ PartitionSet partitions = PartitionSet.create(table().specs());
+ for (DeleteFile file : addedDeleteIndex.referencedDeleteFiles()) {
+ PartitionSpec spec = table().specs().get(file.specId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // an empty set disables partition pruning
+ return PartitionSet.create(table().specs());
+ }
+
+ partitions.add(file.specId(), file.partition());
+ }
+
+ return partitions;
+ }
+
+ /**
+ * Returns the data file locations referenced by the added delete files when every added delete is
+ * file-scoped (a DV or a single-file position delete), or null when some added delete may apply
+ * to files other than the ones it references.
+ */
+ private static Set<String> fileScopedTargets(DeleteFileIndex addedDeleteIndex) {
+ Set<String> locations = Sets.newHashSet();
+ for (DeleteFile file : addedDeleteIndex.referencedDeleteFiles()) {
+ String referencedLocation = ContentFileUtil.referencedDataFileLocation(file);
+ if (referencedLocation == null) {
+ return null;
+ }
+
+ locations.add(referencedLocation);
+ }
+
+ return locations;
+ }
+
+ /** Data file changes observed across the changelog snapshots. */
+ private static class ChangedDataFiles {
+ private final Map<Long, Map<String, ManifestEntry.Status>> statusBySnapshot;
+ private final Set<String> deletedFileLocations;
+ private final PartitionSet deletedFilePartitions;
+
+ ChangedDataFiles(
+ Map<Long, Map<String, ManifestEntry.Status>> statusBySnapshot,
+ Set<String> deletedFileLocations,
+ PartitionSet deletedFilePartitions) {
+ this.statusBySnapshot = statusBySnapshot;
+ this.deletedFileLocations = deletedFileLocations;
+ this.deletedFilePartitions = deletedFilePartitions;
+ }
+
+ static ChangedDataFiles empty(Map<Integer, PartitionSpec> specsById) {
+ return new ChangedDataFiles(Map.of(), Set.of(), PartitionSet.create(specsById));
+ }
+ }
+
+ /** Returns true if any snapshot in the range added or removed a delete file. */
+ private static boolean hasDeleteFileChanges(DeleteFileChanges deleteFileChanges) {
+ for (DeleteFileIndex addedDeletes : deleteFileChanges.addedBySnapshot.values()) {
+ if (!addedDeletes.isEmpty()) {
+ return true;
+ }
+ }
+
+ for (List<DeleteFile> removedDeletes : deleteFileChanges.removedBySnapshot.values()) {
+ if (!removedDeletes.isEmpty()) {
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ /** Returns true if the snapshot the range starts from has any delete files. */
+ private boolean hasExistingDeletes(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return false;
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ return fromSnapshot != null && !fromSnapshot.deleteManifests(table().io()).isEmpty();
+ }
+
+ private boolean includeDeleteFiles() {
+ boolean tableValue =
+ PropertyUtil.propertyAsBoolean(
+ table().properties(),
+ TableProperties.CHANGELOG_SCAN_INCLUDE_DELETE_FILES,
+ TableProperties.CHANGELOG_SCAN_INCLUDE_DELETE_FILES_DEFAULT);
+ return PropertyUtil.propertyAsBoolean(
+ options(), TableProperties.CHANGELOG_SCAN_INCLUDE_DELETE_FILES, tableValue);
+ }
+
+ /**
+ * Collects, for each snapshot, the statuses of data files changed in it, along with the locations
+ * and partitions of data files removed anywhere in the scan range.
+ */
+ private ChangedDataFiles collectChangedDataFiles(Deque<Snapshot> changelogSnapshots) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> statusBySnapshot = Maps.newConcurrentMap();
+ Queue<Set<String>> localDeletedLocations = new ConcurrentLinkedQueue<>();
+ Queue<PartitionSet> localDeletedPartitions = new ConcurrentLinkedQueue<>();
+
+ Tasks.foreach(changelogSnapshots)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .run(
+ snapshot -> {
+ long snapshotId = snapshot.snapshotId();
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+ Set<String> deletedLocations = Sets.newHashSet();
+ PartitionSet deletedPartitions = PartitionSet.create(table().specs());
+
+ List<ManifestFile> changedDataManifests =
+ snapshot.dataManifests(table().io()).stream()
+ .filter(
+ manifest ->
+ manifest.snapshotId() != null && manifest.snapshotId() == snapshotId)
+ .toList();
+
+ if (!changedDataManifests.isEmpty()) {
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (entry.snapshotId() != null && entry.snapshotId() == snapshotId) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ if (entry.status() == ManifestEntry.Status.DELETED) {
+ // copy the file: the reader reuses the entry, so the partition struct
+ // must not be stored as a live reference
+ DataFile deletedFile = entry.file().copyWithoutStats();
+ deletedLocations.add(deletedFile.location());
+ deletedPartitions.add(deletedFile.specId(), deletedFile.partition());
+ }
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to collect file statuses for snapshot " + snapshot.snapshotId(), e);
+ }
+ }
+
+ statusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ localDeletedLocations.add(deletedLocations);
+ localDeletedPartitions.add(deletedPartitions);
+ });
+
+ Set<String> deletedFileLocations = Sets.newHashSet();
+ localDeletedLocations.forEach(deletedFileLocations::addAll);
+
+ PartitionSet deletedFilePartitions = PartitionSet.create(table().specs());
+ localDeletedPartitions.forEach(deletedFilePartitions::addAll);
+
+ return new ChangedDataFiles(statusBySnapshot, deletedFileLocations, deletedFilePartitions);
+ }
+
+ /**
+ * Scope of existing (pre-range) delete files that can apply to changelog tasks in the scan range.
+ * Existing deletes are only attached to data files that were removed in the range or hit by newly
+ * added deletes, so delete files outside this scope can be skipped while building the existing
+ * delete index:
+ *
+ * <ul>
+ * <li>file-scoped position deletes and DVs are kept only if they reference an affected data
+ * file location, or lie in a partition where a partition-scoped delete was added
+ * <li>equality deletes and partition-scoped position deletes are kept only if they have global
+ * reach or lie in an affected partition
+ * </ul>
+ */
+ private static class ExistingDeleteScope {
+ private final Map<Integer, PartitionSpec> specsById;
+ private final PartitionSet allAffectedPartitions;
+ private final PartitionSet broadDeletePartitions;
+ private final Set<String> affectedDataFileLocations;
+
+ ExistingDeleteScope(
+ Map<Integer, PartitionSpec> specsById,
+ PartitionSet allAffectedPartitions,
+ PartitionSet broadDeletePartitions,
+ Set<String> affectedDataFileLocations) {
+ this.specsById = specsById;
+ this.allAffectedPartitions = allAffectedPartitions;
+ this.broadDeletePartitions = broadDeletePartitions;
+ this.affectedDataFileLocations = affectedDataFileLocations;
+ }
+
+ boolean keeps(DeleteFile file) {
+ String referencedLocation = ContentFileUtil.referencedDataFileLocation(file);
+ if (referencedLocation != null) {
+ // file-scoped position delete or DV
+ return affectedDataFileLocations.contains(referencedLocation)
+ || broadDeletePartitions.contains(file.specId(), file.partition());
+ }
+
+ // equality delete or partition-scoped position delete
+ PartitionSpec spec = specsById.get(file.specId());
+ return spec == null
+ || spec.isUnpartitioned()
+ || allAffectedPartitions.contains(file.specId(), file.partition());
+ }
+ }
+
+ /**
+ * Derives the scope of existing deletes that can apply to tasks in the scan range from the
+ * removed data files and the added delete files. Returns null when scoping is not possible
+ * because some added delete has global reach and can affect any data file.
+ */
+ private ExistingDeleteScope buildExistingDeleteScope(
+ ChangedDataFiles changedDataFiles, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ PartitionSet allAffected = PartitionSet.create(table().specs());
+ PartitionSet broadDeletePartitions = PartitionSet.create(table().specs());
+ Set<String> affectedLocations = Sets.newHashSet(changedDataFiles.deletedFileLocations);
+
+ allAffected.addAll(changedDataFiles.deletedFilePartitions);
+
+ for (DeleteFileIndex index : addedDeletesBySnapshot.values()) {
+ for (DeleteFile file : index.referencedDeleteFiles()) {
+ PartitionSpec spec = table().specs().get(file.specId());
+ String referencedLocation = ContentFileUtil.referencedDataFileLocation(file);
+ if (referencedLocation != null) {
+ affectedLocations.add(referencedLocation);
+ } else if (spec == null || spec.isUnpartitioned()) {
+ // an equality delete or a partition-scoped position delete with global reach can
+ // affect any data file, so existing deletes cannot be scoped
+ return null;
+ } else {
+ broadDeletePartitions.add(file.specId(), file.partition());
+ }
+
+ allAffected.add(file.specId(), file.partition());
+ }
+ }
+
+ return new ExistingDeleteScope(
+ table().specs(), allAffected, broadDeletePartitions, affectedLocations);
+ }
+
+ /**
+ * Prunes manifests that cannot contain files in any affected partition. A partitioned manifest
+ * whose spec has no affected partitions is pruned: every delete-to-data match requires equal spec
+ * IDs (file-scoped deletes share their target data file's spec and partition, and
+ * partition-scoped lookups are keyed by the data file's spec), so neither an affected delete file
+ * nor a data file matched by this range's deletes can live in a manifest of an unaffected spec.
+ * Deletes with global reach disable this pruning upstream (null scope for existing deletes; an
+ * empty partition set for per-snapshot data manifest pruning).
+ */
+ private List<ManifestFile> pruneManifestsByAffectedPartitions(
+ List<ManifestFile> manifests, PartitionSet affectedPartitions) {
+ if (affectedPartitions.isEmpty()) {
+ return manifests;
+ }
+
+ Map<Integer, Expression> partitionExprsBySpec =
+ buildAffectedPartitionExpressions(affectedPartitions);
+ if (partitionExprsBySpec == null) {
+ // some affected partition cannot be expressed; skip pruning
+ return manifests;
+ }
+
+ try {
+ Map<Integer, ManifestEvaluator> evaluatorsBySpec = Maps.newHashMap();
+ List<ManifestFile> pruned = Lists.newArrayList();
+ for (ManifestFile manifest : manifests) {
+ PartitionSpec spec = table().specs().get(manifest.partitionSpecId());
+ if (spec == null || spec.isUnpartitioned()) {
+ pruned.add(manifest);
+ continue;
+ }
+
+ Expression expr = partitionExprsBySpec.get(manifest.partitionSpecId());
+ if (expr == null) {
+ // no affected partitions in this manifest's spec: see method javadoc
+ continue;
+ }
+
+ ManifestEvaluator evaluator = evaluatorsBySpec.get(manifest.partitionSpecId());
+ if (evaluator == null) {
+ evaluator = ManifestEvaluator.forPartitionFilter(expr, spec, isCaseSensitive());
+ evaluatorsBySpec.put(manifest.partitionSpecId(), evaluator);
+ }
+
+ if (evaluator.eval(manifest)) {
+ pruned.add(manifest);
+ }
+ }
+
+ return pruned;
+ } catch (Exception e) {
+ // if the partition expressions cannot be evaluated, be conservative and keep all manifests
+ LOG.warn("Failed to prune manifests by affected partitions, skipping pruning", e);
+ return manifests;
+ }
+ }
+
+ /**
+ * Builds partition-space expressions (one per spec) matching the affected partition tuples. The
+ * expressions reference partition field names and compare transformed partition values directly,
+ * so they are correct for non-identity transforms such as bucket or truncate. Returns null if
+ * some affected partition cannot be translated to an expression and pruning must be skipped.
+ */
+ private Map<Integer, Expression> buildAffectedPartitionExpressions(
+ PartitionSet affectedPartitions) {
+ Map<Integer, Expression> exprsBySpec = Maps.newHashMap();
+
+ for (Pair<Integer, StructLike> pair : affectedPartitions) {
+ int specId = pair.first();
+ PartitionSpec spec = table().specs().get(specId);
+ if (spec == null || spec.isUnpartitioned()) {
+ // a partition tuple without a known partitioned spec may affect any manifest
+ return null;
+ }
+
+ StructLike partition = pair.second();
+ Expression tupleExpr = Expressions.alwaysTrue();
+ for (int pos = 0; pos < spec.fields().size(); pos++) {
+ String name = spec.partitionType().fields().get(pos).name();
+ Object value = partition.get(pos, Object.class);
+ if (value == null) {
+ tupleExpr = Expressions.and(tupleExpr, Expressions.isNull(name));
+ } else if (isNaN(value)) {
+ // NaN cannot be expressed as an equality predicate
+ return null;
+ } else {
+ tupleExpr = Expressions.and(tupleExpr, Expressions.equal(name, value));
+ }
+ }
+
+ exprsBySpec.merge(specId, tupleExpr, Expressions::or);
+ }
+
+ return exprsBySpec;
+ }
+
+ private static boolean isNaN(Object value) {
+ return (value instanceof Double && ((Double) value).isNaN())
+ || (value instanceof Float && ((Float) value).isNaN());
+ }
+
+ /**
+ * Tracks the delete files that applied before each snapshot in the scan range and lazily builds a
+ * per-snapshot index of them.
+ *
+ * <p>The index for a snapshot combines deletes that existed before the scan range with deletes
+ * added by earlier snapshots in the range. Delete files that were removed by earlier snapshots in
+ * the range are excluded, so removed delete files are never assigned to tasks.
+ *
+ * <p>Snapshots are consumed oldest to newest, so this class keeps a single advancing cursor over
+ * the range instead of materializing per-snapshot state: one running list of accumulated added
+ * deletes, one running set of removed delete paths, and only the most recently built index.
+ * Within one snapshot, planning worker threads may request the same index concurrently; those
+ * calls are serialized by this object's monitor.
+ */
+ private class CumulativeDeleteIndexes {
+ private final Supplier<DeleteFileIndex> existingDeleteIndexSupplier;
+ private final List<Long> orderedSnapshotIds;
+ private final Map<Long, DeleteFileIndex> addedDeletesBySnapshot;
+ private final Map<Long, List<DeleteFile>> removedDeletesBySnapshot;
+
+ // cursor state: position in orderedSnapshotIds of the next snapshot whose effects to apply,
+ // together with the state produced by all snapshots before that position
+ private int cursor = 0;
+ private final List<DeleteFile> accumulatedAddedDeletes = Lists.newArrayList();
+ private final Set<String> removedPaths = Sets.newHashSet();
+ private Long latestIndexSnapshotId = null;
+ private DeleteFileIndex latestIndex = null;
+
+ CumulativeDeleteIndexes(
+ Deque<Snapshot> snapshots,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Map<Long, List<DeleteFile>> removedDeletesBySnapshot,
+ Supplier<DeleteFileIndex> existingDeleteIndexSupplier) {
+ this.existingDeleteIndexSupplier = existingDeleteIndexSupplier;
+ this.orderedSnapshotIds = snapshots.stream().map(Snapshot::snapshotId).toList();
+ this.addedDeletesBySnapshot = addedDeletesBySnapshot;
+ this.removedDeletesBySnapshot = removedDeletesBySnapshot;
+ }
+
+ /** Returns an index of all delete files that applied before the given snapshot. */
+ synchronized DeleteFileIndex deletesBefore(long snapshotId) {
+ if (latestIndexSnapshotId != null && latestIndexSnapshotId == snapshotId) {
+ return latestIndex;
+ }
+
+ int position = orderedSnapshotIds.indexOf(snapshotId);
+ Preconditions.checkArgument(
+ position >= 0, "Cannot find snapshot %s in the changelog range", snapshotId);
+
+ // a consumer that re-iterates the plan may ask for a snapshot the cursor already passed;
+ // restarting from the beginning is correct, just slower
+ if (position < cursor) {
+ resetCursor();
+ }
+
+ // apply the effects of every snapshot strictly before the requested one
+ while (cursor < position) {
+ advanceCursor(orderedSnapshotIds.get(cursor));
+ cursor += 1;
+ }
+
+ this.latestIndex = buildIndex();
+ this.latestIndexSnapshotId = snapshotId;
+ return latestIndex;
+ }
+
+ private void resetCursor() {
+ this.cursor = 0;
+ accumulatedAddedDeletes.clear();
+ removedPaths.clear();
+ }
+
+ /** Applies one snapshot's delete file removals and additions to the running state. */
+ private void advanceCursor(long snapshotId) {
+ List<DeleteFile> removedDeletes =
+ removedDeletesBySnapshot.getOrDefault(snapshotId, List.of());
+ if (!removedDeletes.isEmpty()) {
+ Set<String> currentRemovedPaths = Sets.newHashSet();
+ for (DeleteFile removed : removedDeletes) {
+ currentRemovedPaths.add(removed.location());
Review Comment:
One DV file might be shared by multiple DVs. It can't removed solely based on location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14264: [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans
URL: https://github.com/apache/iceberg/pull/14264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-3765929660
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-3818585139
Let me reopen and take a look today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2742765660
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +199,526 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Wrapper method that tracks build calls and caches the result for reuse. This ensures we only
+ * build the index once even if called from multiple places.
+ */
+ private DeleteFileIndex buildExistingDeleteIndexTracked(Long fromSnapshotIdExclusive) {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ existingDeleteIndexBuildCallCount++;
+ cachedExistingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
+ return cachedExistingDeleteIndex;
+ }
+
+ // Visible for testing
+ int getExistingDeleteIndexBuildCallCount() {
+ return existingDeleteIndexBuildCallCount;
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount > 0;
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newConcurrentMap();
+ Tasks.foreach(changelogSnapshots)
+ .retry(3)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .onFailure(
+ (snapshot, exc) ->
+ LOG.warn(
+ "Failed to build delete index for snapshot {}", snapshot.snapshotId(), exc))
+ .run(
+ snapshot -> {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ return;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ snapshotDeleteManifests.stream()
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .collect(Collectors.toUnmodifiableList());
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ } else {
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ });
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ DeleteFileIndex cumulativeDeleteIndex = buildDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ // Use a local set per snapshot to track processed files
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ String schemaString = SchemaParser.toJson(schema());
+
+ // Cache per specId - same for all files with same specId
+ Map<Integer, String> specStringCache = Maps.newHashMap();
+ Map<Integer, ResidualEvaluator> residualCache = Maps.newHashMap();
+ Expression residualFilter = shouldIgnoreResiduals() ? Expressions.alwaysTrue() : filter();
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ // Note: alreadyProcessedPaths is local to this snapshot's processing
+ if (alreadyProcessedPaths.contains(filePath)) {
+ continue;
+ }
+
+ // Check if this data file is affected by newly added delete files
+ DeleteFile[] addedDeletes = addedDeleteIndex.forEntry(entry);
+ if (addedDeletes.length == 0) {
+ continue;
+ }
+
+ // This data file was EXISTING but has new delete files applied
+ // Get existing deletes from before this snapshot (cumulative)
+ DeleteFile[] existingDeletes =
+ cumulativeDeleteIndex.isEmpty()
+ ? new DeleteFile[0]
+ : cumulativeDeleteIndex.forEntry(entry);
+
+ // Create a DeletedRowsScanTask
+ int changeOrdinal = snapshotOrdinals.get(snapshot.snapshotId());
+
+ // Use cached values (calculate once per specId)
+ int specId = dataFile.specId();
+ String specString =
+ specStringCache.computeIfAbsent(
+ specId, id -> PartitionSpecParser.toJson(table().specs().get(id)));
+ ResidualEvaluator residuals =
+ residualCache.computeIfAbsent(
+ specId,
+ id -> {
+ PartitionSpec spec = table().specs().get(id);
+ return ResidualEvaluator.of(spec, residualFilter, isCaseSensitive());
+ });
+
+ tasks.add(
+ new BaseDeletedRowsScanTask(
+ changeOrdinal,
+ snapshot.snapshotId(),
+ dataFile.copy(shouldKeepStats()),
+ addedDeletes,
+ existingDeletes,
+ schemaString,
+ specString,
+ residuals));
+
+ // Mark this file as processed for this snapshot
+ alreadyProcessedPaths.add(filePath);
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to plan deleted rows tasks", e);
+ }
+ }
+
+ private boolean shouldKeepStats() {
+ Set<Integer> columns = columnsToKeepStats();
+ return columns != null && !columns.isEmpty();
+ }
+
+ /**
+ * Loads delete files from manifests by parsing each manifest.
+ *
+ * @param manifests the delete manifests to load
+ * @return list of delete files
+ */
+ private Iterable<DeleteFile> loadDeleteFiles(List<ManifestFile> manifests) {
+ Queue<DeleteFile> allDeleteFiles = new ConcurrentLinkedQueue<>();
+
+ Tasks.foreach(manifests)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .run(
+ manifest -> {
+ List<DeleteFile> deleteFiles = loadDeleteFilesFromManifest(manifest);
+ allDeleteFiles.addAll(deleteFiles);
+ });
+
+ return allDeleteFiles;
+ }
+
+ /**
+ * Prunes delete manifests based on partition filter to avoid processing irrelevant manifests.
+ * This significantly improves performance when only a subset of partitions are relevant to the
+ * scan.
+ *
+ * @param manifests all delete manifests to consider
+ * @return list of manifests that might contain relevant delete files
+ */
+ private List<ManifestFile> pruneManifestsByPartition(List<ManifestFile> manifests) {
+ Expression currentFilter = filter();
+
+ // If there's no filter, return all manifests
+ if (currentFilter == null || currentFilter.equals(Expressions.alwaysTrue())) {
+ return manifests;
+ }
+
+ List<ManifestFile> prunedManifests = Lists.newArrayList();
+
+ for (ManifestFile manifest : manifests) {
+ PartitionSpec spec = table().specs().get(manifest.partitionSpecId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // Include unpartitioned manifests
+ prunedManifests.add(manifest);
+ } else if (manifestOverlapsFilter(manifest, spec, currentFilter)) {
+ // Check if manifest partition range overlaps with filter
+ prunedManifests.add(manifest);
+ }
+ }
+
+ return prunedManifests;
+ }
+
+ /**
+ * Checks if a manifest's partition range overlaps with the given filter.
+ *
+ * @param manifest the manifest to check
+ * @param spec the partition spec for the manifest
+ * @param filter the scan filter
+ * @return true if the manifest might contain matching partitions, false otherwise
+ */
+ private boolean manifestOverlapsFilter(
+ ManifestFile manifest, PartitionSpec spec, Expression filter) {
+ try {
+ // Use inclusive projection to transform row filter to partition filter
+ Expression partitionFilter = Projections.inclusive(spec, isCaseSensitive()).project(filter);
+
+ // Create evaluator for the partition filter
+ ManifestEvaluator evaluator =
+ ManifestEvaluator.forPartitionFilter(partitionFilter, spec, isCaseSensitive());
+
+ // Check if manifest could contain matching partitions
+ return evaluator.eval(manifest);
+ } catch (Exception e) {
+ // If evaluation fails, be conservative and include the manifest
+ return true;
+ }
+ }
+
+ /**
+ * Checks if a delete file's partition overlaps with the current scan filter. This enables
+ * partition pruning to reduce memory footprint and planning overhead by skipping delete files
+ * that cannot possibly match any rows in the scan.
+ *
+ * @param file the delete file to check
+ * @return true if the delete file's partition might contain matching rows, false otherwise
+ */
+ private boolean partitionMatchesFilter(DeleteFile file) {
+ // If there's no filter, all partitions match
+ Expression currentFilter = filter();
+ if (currentFilter == null || currentFilter.equals(Expressions.alwaysTrue())) {
+ return true;
+ }
+
+ // Get the partition spec for this delete file
+ PartitionSpec spec = table().specs().get(file.specId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // If spec not found or table is unpartitioned, be conservative and include the file
+ return true;
+ }
+
+ try {
+ // Project the row filter to partition space using inclusive projection
+ // This transforms expressions on source columns to expressions on partition columns
+ Expression partitionFilter =
+ Projections.inclusive(spec, isCaseSensitive()).project(currentFilter);
+
+ // Evaluate the projected filter against the delete file's partition
+ Evaluator evaluator = new Evaluator(spec.partitionType(), partitionFilter, isCaseSensitive());
+ return evaluator.eval(file.partition());
+ } catch (Exception e) {
+ // If evaluation fails, be conservative and include the file
+ return true;
+ }
+ }
+
+ /**
+ * Loads delete files from a single manifest, parsing the manifest entries.
+ *
+ * @param manifest the delete manifest to load
+ * @return list of delete files from this manifest
+ */
+ private List<DeleteFile> loadDeleteFilesFromManifest(ManifestFile manifest) {
+ List<DeleteFile> deleteFiles = Lists.newArrayList();
+
+ try (ManifestReader<DeleteFile> reader =
+ ManifestFiles.readDeleteManifest(manifest, table().io(), table().specs())) {
+ for (ManifestEntry<DeleteFile> entry : reader.entries()) {
+ if (entry.status() != ManifestEntry.Status.DELETED) {
+ // Only include live delete files, copy with minimal stats to save memory
+ DeleteFile file = entry.file();
+
+ // Apply partition pruning - skip delete files that cannot match the scan filter
+ if (!partitionMatchesFilter(file)) {
+ continue;
+ }
+
+ Set<Integer> columns =
+ file.content() == FileContent.POSITION_DELETES
+ ? Set.of(MetadataColumns.DELETE_FILE_PATH.fieldId())
+ : Set.copyOf(file.equalityFieldIds());
+ deleteFiles.add(ContentFileUtil.copy(file, true, columns));
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to read delete manifest: " + manifest.path(), e);
+ }
+
+ return deleteFiles;
+ }
+
private static class CreateDataFileChangeTasks implements CreateTasksFunction<ChangelogScanTask> {
private static final DeleteFile[] NO_DELETES = new DeleteFile[0];
private final Map<Long, Integer> snapshotOrdinals;
-
- CreateDataFileChangeTasks(Deque<Snapshot> snapshots) {
+ private final Supplier<DeleteFileIndex> existingDeleteIndexSupplier;
+ private final Map<Long, DeleteFileIndex> addedDeletesBySnapshot;
+ private final Map<Long, List<DeleteFile>> cumulativeDeletesMap;
Review Comment:
Can we have a situation in which we might assign phantom deletes as we never track delete files that got removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2742750434
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +199,526 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Wrapper method that tracks build calls and caches the result for reuse. This ensures we only
+ * build the index once even if called from multiple places.
+ */
+ private DeleteFileIndex buildExistingDeleteIndexTracked(Long fromSnapshotIdExclusive) {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ existingDeleteIndexBuildCallCount++;
+ cachedExistingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
+ return cachedExistingDeleteIndex;
+ }
+
+ // Visible for testing
+ int getExistingDeleteIndexBuildCallCount() {
+ return existingDeleteIndexBuildCallCount;
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount > 0;
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newConcurrentMap();
+ Tasks.foreach(changelogSnapshots)
+ .retry(3)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .onFailure(
+ (snapshot, exc) ->
+ LOG.warn(
+ "Failed to build delete index for snapshot {}", snapshot.snapshotId(), exc))
+ .run(
+ snapshot -> {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ return;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ snapshotDeleteManifests.stream()
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .collect(Collectors.toUnmodifiableList());
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ } else {
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
Review Comment:
Will this read/index all live delete entries from new delete manifests? How does this handle the case when the new delete manifest has a mix of existing and new delete files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2437500363
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +155,468 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ if (fromSnapshot == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
Review Comment:
I don't believe this behavior is correct (though also not reachable). I believe we validate that the snapshot is present at `BaseIncrementalScan.java:179`. However, if for some reason we got to this point, you shouldn't assume that you start from the beginning of the snapshot ancestry if you cannot find the snapshot. This should be a validation check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2440610373
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
I believe the only scenario where we need this is if there is an equality delete in the incremental scan range. Can't we make this lazy so that we only do this work in the event we encounter an equality delete as opposed to always producing it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445450485
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
+
+ for (DeleteFileIndex addedDeleteIndex : addedDeletesBySnapshot.values()) {
+ if (!addedDeleteIndex.isEmpty()) {
+ // Check if this snapshot has equality deletes
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (df.content() == FileContent.EQUALITY_DELETES) {
+ needsExistingDeleteIndex = true;
+ break;
+ }
+ }
+ if (needsExistingDeleteIndex) {
+ break;
+ }
+ }
+ }
+
+ if (!needsExistingDeleteIndex) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ continue;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ FluentIterable.from(snapshotDeleteManifests)
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(
+ snapshot.snapshotId(), DeleteFileIndex.builderFor(ImmutableList.of()).build());
+ } else {
+ // Load delete files from manifests
+ List<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ }
+
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Process snapshots in order, tracking which files have been handled
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ // Apply partition pruning to only accumulate relevant delete files
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Build cumulative delete index for this snapshot from accumulated deletes
+ DeleteFileIndex cumulativeDeleteIndex = buildCumulativeDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ // Apply partition pruning to only accumulate relevant delete files
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ if (partitionMatchesFilter(df)) {
+ accumulatedDeletes.add(df);
+ }
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a cumulative delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildCumulativeDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ String key = snapshot.snapshotId() + ":" + filePath;
Review Comment:
Why do we need the snapshotId in the key?
Can't this be local for this method and only contain filenames?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-5243188709
Hi @aokolnychyi ,
Sorry i was off last couple weeks. Did you have a chance to complete your review ? If you dont see any issue. Do you think can we merge this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r4078728462
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +270,851 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Manifests that cannot hold a delete relevant to the range are pruned before the index builder
+ * reads the rest, and files outside the affected scope are skipped as entries are read.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, ExistingDeleteScope scope) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> deleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (deleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests that cannot contain deletes for any partition affected by the scan range
+ if (scope != null) {
+ deleteManifests =
+ pruneManifestsByAffectedPartitions(deleteManifests, scope.allAffectedPartitions);
+ }
+
+ if (deleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // filterData prunes manifests and entries against the scan filter with per-spec cached
+ // evaluators and keeps minimal stats. Entry pruning also runs the filter against each delete
+ // file's own stats, which may drop a delete file whose bounds cannot match the filter; that is
+ // only safe because the task residual re-applies the filter to the rows a task emits, so
+ // ignoreResiduals must be forwarded (as ManifestGroup does) to disable it when the residual is
+ // dropped. Deliberately no planWith: this may run lazily under the cumulative-index monitor on
+ // shared worker-pool threads, and nested submission to the same pool can starve it
+ DeleteFileIndex.Builder builder =
+ DeleteFileIndex.builderFor(table().io(), deleteManifests)
+ .specsById(table().specs())
Review Comment:
` .schemasById(table.schemas())` is needed as well to cover columns used by equality deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-3797455727
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2417576263
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
@pvary I'm not sure the case you provide is accurate because we would only be producing changes for S3, not snapshots prior to it. If this is incremental, it should only be the observed changes within the range, not changes prior to it. Since deletes only affect prior data, it would have no effect on the results of the scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2419065337
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -71,6 +80,12 @@ protected CloseableIterable<ChangelogScanTask> doPlanFiles(
.filter(manifest -> changelogSnapshotIds.contains(manifest.snapshotId()))
.toSet();
+ // Build delete file index for existing deletes (before the start snapshot)
+ DeleteFileIndex existingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
Review Comment:
@talatuyarer: Don't forget about overlapping equality deletes too.
Flink could generate the following:
- Snapshot 1 - Insert a record (data file (D1) added with PK1)
- Snapshot 2 - Update a record (equality delete file (EQ1) added with PK1, and data file (D2) added with PK1)
- Snapshot 3 - Update a record again (equality delete file (EQ2) added with PK1, and data file (D3) added with PK1)
When we are emitting rows for Snapshot 3:
- We need to scan D1 but should not emit a delete record for PK1 as it was already deleted in Snapshot 2
- We need to scan D2 and emit a delete record for PK1 as it was not deleted in Snapshot 2
- We need to scan D3 and emit the data record for PK1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2440548486
##########
core/src/main/java/org/apache/iceberg/DeleteManifestCache.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.TimeUnit;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+
+/**
+ * Cache for parsed delete file entries from manifests to avoid redundant manifest parsing across
+ * changelog scans.
+ *
+ * <p>This cache is thread-safe and uses LRU eviction with size and time-based expiration.
+ */
+class DeleteManifestCache {
Review Comment:
I'm somewhat sceptical of using a cache here. 1k entries is really only good for smaller cases and once you've exhausted that, the value of the cache degrates rapidly. I'm a little concerned that at larger scales the cache will be completely ineffective as entries would just cycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2437500363
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +155,468 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ if (fromSnapshot == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
Review Comment:
I don't believe this behavior is correct (though also not reachable). I believe we validate that the snapshot is present at `BaseIncrementalScan.java:179`. However, if for some reason we got to this point, you shouldn't assume that you start from the beginning of the snapshot ancestry if you cannot find the snapshot. This should be a precondition check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445460290
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -179,5 +673,40 @@ public CloseableIterable<ChangelogScanTask> apply(
}
});
}
+
+ /**
+ * Gets all delete files that apply to an ADDED data file. This includes existing deletes that
+ * were present before this snapshot and any new deletes added in this snapshot.
+ */
+ private DeleteFile[] getDeletesForAddedFile(
+ ManifestEntry<DataFile> entry, long commitSnapshotId) {
+ DeleteFile[] existingDeletes =
+ existingDeleteIndex.isEmpty() ? NO_DELETES : existingDeleteIndex.forEntry(entry);
Review Comment:
I don't see how could this happen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2445458227
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -155,21 +640,30 @@ public CloseableIterable<ChangelogScanTask> apply(
switch (entry.status()) {
case ADDED:
+ // For ADDED data files, attach any delete files that apply to them
+ // This includes both existing deletes and newly added deletes in this snapshot
Review Comment:
existing deletes should not affect newly added files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2483064014
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -179,5 +673,40 @@ public CloseableIterable<ChangelogScanTask> apply(
}
});
}
+
+ /**
+ * Gets all delete files that apply to an ADDED data file. This includes existing deletes that
+ * were present before this snapshot and any new deletes added in this snapshot.
+ */
+ private DeleteFile[] getDeletesForAddedFile(
+ ManifestEntry<DataFile> entry, long commitSnapshotId) {
+ DeleteFile[] existingDeletes =
+ existingDeleteIndex.isEmpty() ? NO_DELETES : existingDeleteIndex.forEntry(entry);
Review Comment:
I can delete specific rows before deleting whole file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#issuecomment-3981394313
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2742880969
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +199,526 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Wrapper method that tracks build calls and caches the result for reuse. This ensures we only
+ * build the index once even if called from multiple places.
+ */
+ private DeleteFileIndex buildExistingDeleteIndexTracked(Long fromSnapshotIdExclusive) {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ existingDeleteIndexBuildCallCount++;
+ cachedExistingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
+ return cachedExistingDeleteIndex;
+ }
+
+ // Visible for testing
+ int getExistingDeleteIndexBuildCallCount() {
+ return existingDeleteIndexBuildCallCount;
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount > 0;
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newConcurrentMap();
+ Tasks.foreach(changelogSnapshots)
+ .retry(3)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .onFailure(
+ (snapshot, exc) ->
+ LOG.warn(
+ "Failed to build delete index for snapshot {}", snapshot.snapshotId(), exc))
+ .run(
+ snapshot -> {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ return;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ snapshotDeleteManifests.stream()
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .collect(Collectors.toUnmodifiableList());
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ } else {
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ });
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ DeleteFileIndex cumulativeDeleteIndex = buildDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ // Use a local set per snapshot to track processed files
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to collect file statuses", e);
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ }
+
+ return fileStatusBySnapshot;
+ }
+
+ /** Builds a delete index from the accumulated list of delete files. */
+ private DeleteFileIndex buildDeleteIndex(List<DeleteFile> accumulatedDeletes) {
+ if (accumulatedDeletes.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ return DeleteFileIndex.builderFor(accumulatedDeletes)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Processes data files for a snapshot to create DeletedRowsScanTask for existing files affected
+ * by new delete files.
+ */
+ private void processSnapshotForDeletedRowsTasks(
+ Snapshot snapshot,
+ DeleteFileIndex addedDeleteIndex,
+ DeleteFileIndex cumulativeDeleteIndex,
+ Map<String, ManifestEntry.Status> currentSnapshotFiles,
+ Set<String> alreadyProcessedPaths,
+ Map<Long, Integer> snapshotOrdinals,
+ List<ChangelogScanTask> tasks) {
+
+ // Get all data files that exist in this snapshot
+ List<ManifestFile> allDataManifests = snapshot.dataManifests(table().io());
+ ManifestGroup allDataGroup =
+ new ManifestGroup(table().io(), allDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreDeleted()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ if (shouldIgnoreResiduals()) {
+ allDataGroup = allDataGroup.ignoreResiduals();
+ }
+
+ String schemaString = SchemaParser.toJson(schema());
+
+ // Cache per specId - same for all files with same specId
+ Map<Integer, String> specStringCache = Maps.newHashMap();
+ Map<Integer, ResidualEvaluator> residualCache = Maps.newHashMap();
+ Expression residualFilter = shouldIgnoreResiduals() ? Expressions.alwaysTrue() : filter();
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = allDataGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ DataFile dataFile = entry.file();
+ String filePath = dataFile.location();
+
+ // Skip if this file was ADDED or DELETED in this snapshot
+ // (those are handled by CreateDataFileChangeTasks)
+ if (currentSnapshotFiles.containsKey(filePath)) {
+ continue;
+ }
+
+ // Skip if we already created a task for this file in this snapshot
+ // Note: alreadyProcessedPaths is local to this snapshot's processing
+ if (alreadyProcessedPaths.contains(filePath)) {
+ continue;
+ }
+
+ // Check if this data file is affected by newly added delete files
+ DeleteFile[] addedDeletes = addedDeleteIndex.forEntry(entry);
+ if (addedDeletes.length == 0) {
+ continue;
+ }
+
+ // This data file was EXISTING but has new delete files applied
+ // Get existing deletes from before this snapshot (cumulative)
+ DeleteFile[] existingDeletes =
+ cumulativeDeleteIndex.isEmpty()
+ ? new DeleteFile[0]
+ : cumulativeDeleteIndex.forEntry(entry);
+
+ // Create a DeletedRowsScanTask
+ int changeOrdinal = snapshotOrdinals.get(snapshot.snapshotId());
+
+ // Use cached values (calculate once per specId)
+ int specId = dataFile.specId();
+ String specString =
+ specStringCache.computeIfAbsent(
+ specId, id -> PartitionSpecParser.toJson(table().specs().get(id)));
+ ResidualEvaluator residuals =
+ residualCache.computeIfAbsent(
+ specId,
+ id -> {
+ PartitionSpec spec = table().specs().get(id);
+ return ResidualEvaluator.of(spec, residualFilter, isCaseSensitive());
+ });
+
+ tasks.add(
+ new BaseDeletedRowsScanTask(
+ changeOrdinal,
+ snapshot.snapshotId(),
+ dataFile.copy(shouldKeepStats()),
+ addedDeletes,
+ existingDeletes,
+ schemaString,
+ specString,
+ residuals));
+
+ // Mark this file as processed for this snapshot
+ alreadyProcessedPaths.add(filePath);
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to plan deleted rows tasks", e);
+ }
+ }
+
+ private boolean shouldKeepStats() {
+ Set<Integer> columns = columnsToKeepStats();
+ return columns != null && !columns.isEmpty();
+ }
+
+ /**
+ * Loads delete files from manifests by parsing each manifest.
+ *
+ * @param manifests the delete manifests to load
+ * @return list of delete files
+ */
+ private Iterable<DeleteFile> loadDeleteFiles(List<ManifestFile> manifests) {
+ Queue<DeleteFile> allDeleteFiles = new ConcurrentLinkedQueue<>();
+
+ Tasks.foreach(manifests)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .run(
+ manifest -> {
+ List<DeleteFile> deleteFiles = loadDeleteFilesFromManifest(manifest);
+ allDeleteFiles.addAll(deleteFiles);
+ });
+
+ return allDeleteFiles;
+ }
+
+ /**
+ * Prunes delete manifests based on partition filter to avoid processing irrelevant manifests.
+ * This significantly improves performance when only a subset of partitions are relevant to the
+ * scan.
+ *
+ * @param manifests all delete manifests to consider
+ * @return list of manifests that might contain relevant delete files
+ */
+ private List<ManifestFile> pruneManifestsByPartition(List<ManifestFile> manifests) {
+ Expression currentFilter = filter();
+
+ // If there's no filter, return all manifests
+ if (currentFilter == null || currentFilter.equals(Expressions.alwaysTrue())) {
+ return manifests;
+ }
+
+ List<ManifestFile> prunedManifests = Lists.newArrayList();
+
+ for (ManifestFile manifest : manifests) {
+ PartitionSpec spec = table().specs().get(manifest.partitionSpecId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // Include unpartitioned manifests
+ prunedManifests.add(manifest);
+ } else if (manifestOverlapsFilter(manifest, spec, currentFilter)) {
+ // Check if manifest partition range overlaps with filter
+ prunedManifests.add(manifest);
+ }
+ }
+
+ return prunedManifests;
+ }
+
+ /**
+ * Checks if a manifest's partition range overlaps with the given filter.
+ *
+ * @param manifest the manifest to check
+ * @param spec the partition spec for the manifest
+ * @param filter the scan filter
+ * @return true if the manifest might contain matching partitions, false otherwise
+ */
+ private boolean manifestOverlapsFilter(
+ ManifestFile manifest, PartitionSpec spec, Expression filter) {
+ try {
+ // Use inclusive projection to transform row filter to partition filter
+ Expression partitionFilter = Projections.inclusive(spec, isCaseSensitive()).project(filter);
+
+ // Create evaluator for the partition filter
+ ManifestEvaluator evaluator =
+ ManifestEvaluator.forPartitionFilter(partitionFilter, spec, isCaseSensitive());
+
+ // Check if manifest could contain matching partitions
+ return evaluator.eval(manifest);
+ } catch (Exception e) {
+ // If evaluation fails, be conservative and include the manifest
+ return true;
+ }
+ }
+
+ /**
+ * Checks if a delete file's partition overlaps with the current scan filter. This enables
+ * partition pruning to reduce memory footprint and planning overhead by skipping delete files
+ * that cannot possibly match any rows in the scan.
+ *
+ * @param file the delete file to check
+ * @return true if the delete file's partition might contain matching rows, false otherwise
+ */
+ private boolean partitionMatchesFilter(DeleteFile file) {
+ // If there's no filter, all partitions match
+ Expression currentFilter = filter();
+ if (currentFilter == null || currentFilter.equals(Expressions.alwaysTrue())) {
+ return true;
+ }
+
+ // Get the partition spec for this delete file
+ PartitionSpec spec = table().specs().get(file.specId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // If spec not found or table is unpartitioned, be conservative and include the file
+ return true;
+ }
+
+ try {
+ // Project the row filter to partition space using inclusive projection
+ // This transforms expressions on source columns to expressions on partition columns
+ Expression partitionFilter =
+ Projections.inclusive(spec, isCaseSensitive()).project(currentFilter);
+
+ // Evaluate the projected filter against the delete file's partition
+ Evaluator evaluator = new Evaluator(spec.partitionType(), partitionFilter, isCaseSensitive());
+ return evaluator.eval(file.partition());
+ } catch (Exception e) {
+ // If evaluation fails, be conservative and include the file
+ return true;
+ }
+ }
+
+ /**
+ * Loads delete files from a single manifest, parsing the manifest entries.
+ *
+ * @param manifest the delete manifest to load
+ * @return list of delete files from this manifest
+ */
+ private List<DeleteFile> loadDeleteFilesFromManifest(ManifestFile manifest) {
+ List<DeleteFile> deleteFiles = Lists.newArrayList();
+
+ try (ManifestReader<DeleteFile> reader =
+ ManifestFiles.readDeleteManifest(manifest, table().io(), table().specs())) {
+ for (ManifestEntry<DeleteFile> entry : reader.entries()) {
+ if (entry.status() != ManifestEntry.Status.DELETED) {
+ // Only include live delete files, copy with minimal stats to save memory
+ DeleteFile file = entry.file();
+
+ // Apply partition pruning - skip delete files that cannot match the scan filter
+ if (!partitionMatchesFilter(file)) {
+ continue;
+ }
+
+ Set<Integer> columns =
+ file.content() == FileContent.POSITION_DELETES
+ ? Set.of(MetadataColumns.DELETE_FILE_PATH.fieldId())
+ : Set.copyOf(file.equalityFieldIds());
+ deleteFiles.add(ContentFileUtil.copy(file, true, columns));
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to read delete manifest: " + manifest.path(), e);
+ }
+
+ return deleteFiles;
+ }
+
private static class CreateDataFileChangeTasks implements CreateTasksFunction<ChangelogScanTask> {
private static final DeleteFile[] NO_DELETES = new DeleteFile[0];
private final Map<Long, Integer> snapshotOrdinals;
-
- CreateDataFileChangeTasks(Deque<Snapshot> snapshots) {
+ private final Supplier<DeleteFileIndex> existingDeleteIndexSupplier;
+ private final Map<Long, DeleteFileIndex> addedDeletesBySnapshot;
+ private final Map<Long, List<DeleteFile>> cumulativeDeletesMap;
Review Comment:
Say we have an operation that rewrites an equality delete file into a position delete file and replaces the deletes in the manifests. Given that we didn't track the equality delete file was removed, we may still assign it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2742758269
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +199,526 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Wrapper method that tracks build calls and caches the result for reuse. This ensures we only
+ * build the index once even if called from multiple places.
+ */
+ private DeleteFileIndex buildExistingDeleteIndexTracked(Long fromSnapshotIdExclusive) {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ existingDeleteIndexBuildCallCount++;
+ cachedExistingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
+ return cachedExistingDeleteIndex;
+ }
+
+ // Visible for testing
+ int getExistingDeleteIndexBuildCallCount() {
+ return existingDeleteIndexBuildCallCount;
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount > 0;
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newConcurrentMap();
+ Tasks.foreach(changelogSnapshots)
+ .retry(3)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .onFailure(
+ (snapshot, exc) ->
+ LOG.warn(
+ "Failed to build delete index for snapshot {}", snapshot.snapshotId(), exc))
+ .run(
+ snapshot -> {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ return;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ snapshotDeleteManifests.stream()
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .collect(Collectors.toUnmodifiableList());
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ } else {
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(addedDeleteManifests);
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ });
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses for each snapshot
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ DeleteFileIndex cumulativeDeleteIndex = buildDeleteIndex(accumulatedDeletes);
+
+ // Process data files for this snapshot
+ // Use a local set per snapshot to track processed files
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Map<Long, Map<String, ManifestEntry.Status>> buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newHashMap();
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+
+ List<ManifestFile> changedDataManifests =
Review Comment:
Shouldn't we scan all data manifest for potentially impacted files when a new delete gets added?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2483064014
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -179,5 +673,40 @@ public CloseableIterable<ChangelogScanTask> apply(
}
});
}
+
+ /**
+ * Gets all delete files that apply to an ADDED data file. This includes existing deletes that
+ * were present before this snapshot and any new deletes added in this snapshot.
+ */
+ private DeleteFile[] getDeletesForAddedFile(
+ ManifestEntry<DataFile> entry, long commitSnapshotId) {
+ DeleteFile[] existingDeletes =
+ existingDeleteIndex.isEmpty() ? NO_DELETES : existingDeleteIndex.forEntry(entry);
Review Comment:
I can delete specific rows before deleting whole file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r2483125421
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +158,473 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.builderFor(ImmutableList.of()).build();
+ }
+
+ // Check if we need existingDeleteIndex for equality deletes
+ boolean needsExistingDeleteIndex = false;
Review Comment:
I implemented lazyIndexbuild as you suggested and also added 5 tests to make sure it works as expected. If there is any equaity delete on my scan range. I call buildExistingDeleteIndex, if there is no equality delete on my delete files. I lazyly buildExistingDeleteIndex whenever I see DeleteFile. If I have built the index for equality delete, I reused for DeleteFile.
No second pass no redundant index build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r3464569536
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +205,701 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests, null);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Wrapper method that tracks build calls and caches the result for reuse. This ensures we only
+ * build the index once even if called from multiple places.
+ */
+ private DeleteFileIndex buildExistingDeleteIndexTracked(Long fromSnapshotIdExclusive) {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ existingDeleteIndexBuildCallCount++;
+ cachedExistingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
+ return cachedExistingDeleteIndex;
+ }
+
+ // Visible for testing
+ int getExistingDeleteIndexBuildCallCount() {
+ return existingDeleteIndexBuildCallCount;
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount > 0;
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newConcurrentMap();
+ Tasks.foreach(changelogSnapshots)
+ .retry(3)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .onFailure(
+ (snapshot, exc) ->
+ LOG.warn(
+ "Failed to build delete index for snapshot {}", snapshot.snapshotId(), exc))
+ .run(
+ snapshot -> {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ return;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ snapshotDeleteManifests.stream()
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .collect(Collectors.toUnmodifiableList());
Review Comment:
can be replaced with 'toList()'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core] Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r3426826670
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +205,701 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Uses manifest pruning and caching to optimize performance.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> existingDeleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (existingDeleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests based on partition filter to avoid processing irrelevant manifests
+ List<ManifestFile> prunedManifests = pruneManifestsByPartition(existingDeleteManifests);
+ if (prunedManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles = loadDeleteFiles(prunedManifests, null);
+
+ return DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ }
+
+ /**
+ * Wrapper method that tracks build calls and caches the result for reuse. This ensures we only
+ * build the index once even if called from multiple places.
+ */
+ private DeleteFileIndex buildExistingDeleteIndexTracked(Long fromSnapshotIdExclusive) {
+ if (cachedExistingDeleteIndex != null) {
+ return cachedExistingDeleteIndex;
+ }
+ existingDeleteIndexBuildCallCount++;
+ cachedExistingDeleteIndex = buildExistingDeleteIndex(fromSnapshotIdExclusive);
+ return cachedExistingDeleteIndex;
+ }
+
+ // Visible for testing
+ int getExistingDeleteIndexBuildCallCount() {
+ return existingDeleteIndexBuildCallCount;
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount > 0;
+ }
+
+ /**
+ * Builds per-snapshot delete file indexes for newly added delete files in each changelog
+ * snapshot. These deletes should generate DELETE changelog rows. Uses caching to avoid re-parsing
+ * manifests.
+ */
+ private Map<Long, DeleteFileIndex> buildAddedDeleteIndexes(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot = Maps.newConcurrentMap();
+ Tasks.foreach(changelogSnapshots)
+ .retry(3)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .onFailure(
+ (snapshot, exc) ->
+ LOG.warn(
+ "Failed to build delete index for snapshot {}", snapshot.snapshotId(), exc))
+ .run(
+ snapshot -> {
+ List<ManifestFile> snapshotDeleteManifests = snapshot.deleteManifests(table().io());
+ if (snapshotDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ return;
+ }
+
+ // Filter to only include delete files added in this snapshot
+ List<ManifestFile> addedDeleteManifests =
+ snapshotDeleteManifests.stream()
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .collect(Collectors.toUnmodifiableList());
+
+ if (addedDeleteManifests.isEmpty()) {
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), DeleteFileIndex.emptyIndex());
+ } else {
+ // Load delete files from manifests
+ Iterable<DeleteFile> deleteFiles =
+ loadDeleteFiles(addedDeleteManifests, snapshot.snapshotId());
+
+ DeleteFileIndex index =
+ DeleteFileIndex.builderFor(deleteFiles)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build();
+ addedDeletesBySnapshot.put(snapshot.snapshotId(), index);
+ }
+ });
+ return addedDeletesBySnapshot;
+ }
+
+ /**
+ * Plans tasks for EXISTING data files that are affected by newly added delete files. These files
+ * were not added or deleted in the changelog snapshot range, but have new delete files applied to
+ * them.
+ */
+ private CloseableIterable<ChangelogScanTask> planDeletedRowsTasks(
+ Deque<Snapshot> changelogSnapshots,
+ DeleteFileIndex existingDeleteIndex,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Integer> snapshotOrdinals = computeSnapshotOrdinals(changelogSnapshots);
+ List<ChangelogScanTask> tasks = Lists.newArrayList();
+
+ // Build a map of file statuses and collect affected partitions for each snapshot
+ Pair<Map<Long, Map<String, ManifestEntry.Status>>, PartitionSet> fileStatusAndPartitions =
+ buildFileStatusBySnapshot(changelogSnapshots, changelogSnapshotIds);
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot =
+ fileStatusAndPartitions.first();
+ PartitionSet affectedPartitions = fileStatusAndPartitions.second();
+
+ // Accumulate actual DeleteFile entries chronologically
+ List<DeleteFile> accumulatedDeletes = Lists.newArrayList();
+
+ // Start with deletes from before the changelog range
+ if (!existingDeleteIndex.isEmpty()) {
+ for (DeleteFile df : existingDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ for (Snapshot snapshot : changelogSnapshots) {
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshot.snapshotId());
+ if (addedDeleteIndex.isEmpty()) {
+ continue;
+ }
+
+ // Collect partitions of newly added delete files for pruning (important for the current
+ // snapshot)
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ affectedPartitions.add(df.specId(), df.partition());
+ }
+
+ DeleteFileIndex cumulativeDeleteIndex =
+ buildDeleteIndex(accumulatedDeletes, affectedPartitions);
+
+ // Process data files for this snapshot
+ // Use a local set per snapshot to track processed files
+ Set<String> alreadyProcessedPaths = Sets.newHashSet();
+ processSnapshotForDeletedRowsTasks(
+ snapshot,
+ addedDeleteIndex,
+ cumulativeDeleteIndex,
+ fileStatusBySnapshot.get(snapshot.snapshotId()),
+ alreadyProcessedPaths,
+ snapshotOrdinals,
+ affectedPartitions,
+ tasks);
+
+ // Accumulate this snapshot's added deletes for subsequent snapshots
+ for (DeleteFile df : addedDeleteIndex.referencedDeleteFiles()) {
+ accumulatedDeletes.add(df);
+ }
+ }
+
+ return CloseableIterable.withNoopClose(tasks);
+ }
+
+ /**
+ * Builds a map of file statuses for each snapshot, tracking which files were added or deleted in
+ * each snapshot.
+ */
+ private Pair<Map<Long, Map<String, ManifestEntry.Status>>, PartitionSet>
+ buildFileStatusBySnapshot(
+ Deque<Snapshot> changelogSnapshots, Set<Long> changelogSnapshotIds) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> fileStatusBySnapshot = Maps.newConcurrentMap();
+ java.util.Queue<PartitionSet> localPartitionsQueue =
+ new java.util.concurrent.ConcurrentLinkedQueue<>();
+
+ Tasks.foreach(changelogSnapshots)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .run(
+ snapshot -> {
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+ PartitionSet localAffected = PartitionSet.create(table().specs());
+
+ List<ManifestFile> changedDataManifests =
+ FluentIterable.from(snapshot.dataManifests(table().io()))
+ .filter(manifest -> manifest.snapshotId().equals(snapshot.snapshotId()))
+ .toList();
+
+ if (!changedDataManifests.isEmpty()) {
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (changelogSnapshotIds.contains(entry.snapshotId())) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ localAffected.add(entry.file().specId(), entry.file().partition());
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to collect file statuses for snapshot " + snapshot.snapshotId(), e);
+ }
+ }
+
+ fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ localPartitionsQueue.add(localAffected);
+ });
+
+ PartitionSet globalAffected = PartitionSet.create(table().specs());
+ for (PartitionSet local : localPartitionsQueue) {
+ globalAffected.addAll(local);
+ }
+
+ return Pair.of(fileStatusBySnapshot, globalAffected);
+ }
+
+ private List<ManifestFile> pruneManifestsByAffectedPartitions(
+ List<ManifestFile> manifests, PartitionSet affectedPartitions) {
+ if (affectedPartitions.isEmpty()) {
+ return manifests;
+ }
+
+ Expression affectedExpr = buildAffectedPartitionExpression(affectedPartitions);
+ if (affectedExpr == Expressions.alwaysFalse()) {
+ return manifests;
+ }
+
+ List<ManifestFile> pruned = Lists.newArrayList();
+ for (ManifestFile manifest : manifests) {
+ PartitionSpec spec = table().specs().get(manifest.partitionSpecId());
+ if (spec == null || spec.isUnpartitioned()) {
+ pruned.add(manifest);
+ } else if (manifestOverlapsFilter(manifest, spec, affectedExpr)) {
+ pruned.add(manifest);
+ }
+ }
+ return pruned;
+ }
+
+ private Expression buildAffectedPartitionExpression(PartitionSet affectedPartitions) {
+ Expression combined = null;
+
+ for (Pair<Integer, StructLike> pair : affectedPartitions) {
+ int specId = pair.first();
+ StructLike partition = pair.second();
+ PartitionSpec spec = table().specs().get(specId);
+ if (spec == null) {
+ continue;
+ } else if (spec.isUnpartitioned()) {
+ return Expressions.alwaysTrue(); // FALLBACK: Global delete exists, include ALL manifests!
+ }
+
+ Expression specExpr = null;
+ for (int i = 0; i < spec.fields().size(); i++) {
+ org.apache.iceberg.PartitionField field = spec.fields().get(i);
+ Object value = partition.get(i, Object.class);
+ if (value != null) {
+ String columnName = table().schema().findColumnName(field.sourceId());
+ if (columnName != null) {
+ Expression equalExpr = Expressions.equal(columnName, value);
Review Comment:
this is not correct for bucket transform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add Delete File Support and Partition Pruning for Incremental Changelog Scans [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r4079005758
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +270,851 @@ private static Map<Long, Integer> computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before the start snapshot.
+ * These deletes should be applied to data files but should not generate DELETE changelog rows.
+ * Manifests that cannot hold a delete relevant to the range are pruned before the index builder
+ * reads the rest, and files outside the affected scope are skipped as entries are read.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, ExistingDeleteScope scope) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s", fromSnapshotIdExclusive);
+
+ List<ManifestFile> deleteManifests = fromSnapshot.deleteManifests(table().io());
+ if (deleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests that cannot contain deletes for any partition affected by the scan range
+ if (scope != null) {
+ deleteManifests =
+ pruneManifestsByAffectedPartitions(deleteManifests, scope.allAffectedPartitions);
+ }
+
+ if (deleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // filterData prunes manifests and entries against the scan filter with per-spec cached
+ // evaluators and keeps minimal stats. Entry pruning also runs the filter against each delete
+ // file's own stats, which may drop a delete file whose bounds cannot match the filter; that is
+ // only safe because the task residual re-applies the filter to the rows a task emits, so
+ // ignoreResiduals must be forwarded (as ManifestGroup does) to disable it when the residual is
+ // dropped. Deliberately no planWith: this may run lazily under the cumulative-index monitor on
+ // shared worker-pool threads, and nested submission to the same pool can starve it
+ DeleteFileIndex.Builder builder =
+ DeleteFileIndex.builderFor(table().io(), deleteManifests)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .filterData(filter());
+
+ if (shouldIgnoreResiduals()) {
+ builder.ignoreResiduals();
+ }
+
+ if (scope != null) {
+ // skip delete files that cannot apply to any task in the scan range
+ builder.deleteFilePredicate(scope::keeps);
+ }
+
+ return builder.build();
+ }
+
+ /**
+ * Builds the existing (pre-range) delete index at most once per planFiles() invocation, so that
+ * concurrent planning threads share one index while a later plan of the same scan, which may
+ * resolve a different range and scope, builds its own.
+ */
+ private class ExistingDeleteIndexHolder {
+ private final Long fromSnapshotIdExclusive;
+ private final ExistingDeleteScope scope;
+ private DeleteFileIndex index = null;
+
+ ExistingDeleteIndexHolder(Long fromSnapshotIdExclusive, ExistingDeleteScope scope) {
+ this.fromSnapshotIdExclusive = fromSnapshotIdExclusive;
+ this.scope = scope;
+ }
+
+ synchronized DeleteFileIndex get() {
+ if (index == null) {
+ this.index = buildExistingDeleteIndex(fromSnapshotIdExclusive, scope);
+ existingDeleteIndexBuildCallCount.incrementAndGet();
+ }
+
+ return index;
+ }
+ }
+
+ // Visible for testing
+ int existingDeleteIndexBuildCount() {
+ return existingDeleteIndexBuildCallCount.get();
+ }
+
+ // Visible for testing
+ boolean wasExistingDeleteIndexBuilt() {
+ return existingDeleteIndexBuildCallCount.get() > 0;
+ }
+
+ /** Added delete indexes and removed delete files, per snapshot, read in one manifest pass. */
+ private static class DeleteFileChanges {
+ private final Map<Long, DeleteFileIndex> addedBySnapshot;
+ private final Map<Long, List<DeleteFile>> removedBySnapshot;
+
+ DeleteFileChanges(
+ Map<Long, DeleteFileIndex> addedBySnapshot, Map<Long, List<DeleteFile>> removedBySnapshot) {
+ this.addedBySnapshot = addedBySnapshot;
+ this.removedBySnapshot = removedBySnapshot;
+ }
+ }
+
+ /**
+ * Reads the delete manifests written by each changelog snapshot exactly once, splitting live
+ * entries into the snapshot's added delete files (indexed) and removed delete files.
+ */
+ private DeleteFileChanges collectDeleteFileChanges(Deque<Snapshot> changelogSnapshots) {
+ Map<Long, DeleteFileIndex> addedBySnapshot = Maps.newConcurrentMap();
+ Map<Long, List<DeleteFile>> removedBySnapshot = Maps.newConcurrentMap();
+
+ // one evaluator of the scan filter per partition spec, shared by all manifests and snapshots
+ Map<Integer, Evaluator> partitionEvaluators = Maps.newConcurrentMap();
+
+ Tasks.foreach(changelogSnapshots)
+ .retry(3)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .onFailure(
+ (snapshot, exc) ->
+ LOG.warn(
+ "Failed to read delete manifests for snapshot {}", snapshot.snapshotId(), exc))
+ .run(
+ snapshot -> {
+ long snapshotId = snapshot.snapshotId();
+ List<ManifestFile> newDeleteManifests =
+ snapshot.deleteManifests(table().io()).stream()
+ .filter(
+ manifest ->
+ manifest.snapshotId() != null && manifest.snapshotId() == snapshotId)
+ .toList();
+
+ List<DeleteFile> added = Lists.newArrayList();
+ List<DeleteFile> removed = Lists.newArrayList();
+ for (ManifestFile manifest : newDeleteManifests) {
+ readDeleteManifest(manifest, snapshotId, partitionEvaluators, added, removed);
+ }
+
+ addedBySnapshot.put(
+ snapshotId,
+ added.isEmpty()
+ ? DeleteFileIndex.emptyIndex()
+ : DeleteFileIndex.builderFor(added)
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .build());
+ removedBySnapshot.put(snapshotId, removed);
+ });
+
+ return new DeleteFileChanges(addedBySnapshot, removedBySnapshot);
+ }
+
+ /**
+ * Reads one delete manifest, collecting the delete files this snapshot added and removed. Only
+ * entries committed by the given snapshot are considered; EXISTING entries carried over from
+ * earlier snapshots keep their original snapshot ID and are skipped.
+ */
+ private void readDeleteManifest(
+ ManifestFile manifest,
+ long snapshotId,
+ Map<Integer, Evaluator> partitionEvaluators,
+ List<DeleteFile> added,
+ List<DeleteFile> removed) {
+ try (ManifestReader<DeleteFile> reader =
+ ManifestFiles.readDeleteManifest(manifest, table().io(), table().specs())) {
+ for (ManifestEntry<DeleteFile> entry : reader.entries()) {
+ if (entry.snapshotId() == null || entry.snapshotId() != snapshotId) {
+ continue;
+ }
+
+ DeleteFile file = entry.file();
+
+ // Apply partition pruning - skip delete files that cannot match the scan filter
+ if (!partitionMatchesFilter(file, partitionEvaluators)) {
+ continue;
+ }
+
+ Set<Integer> columns =
+ file.content() == FileContent.POSITION_DELETES
+ ? Set.of(MetadataColumns.DELETE_FILE_PATH.fieldId())
+ : Set.copyOf(file.equalityFieldIds());
+ DeleteFile copied = ContentFileUtil.copy(file, true, columns);
+
+ if (entry.status() == ManifestEntry.Status.DELETED) {
+ removed.add(copied);
+ } else {
+ added.add(copied);
+ }
+ }
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to read delete manifest: " + manifest.path(), e);
+ }
+ }
+
+ /**
+ * Returns true if the delete file's partition may contain rows matching the scan filter, so that
+ * delete files that cannot apply to any scanned row are skipped. The scan filter is projected
+ * into each spec's partition space once and the resulting evaluator is cached in the given map,
+ * which may be shared by concurrent readers.
+ */
+ private boolean partitionMatchesFilter(DeleteFile file, Map<Integer, Evaluator> evaluatorCache) {
+ Expression currentFilter = filter();
+ if (currentFilter.equals(Expressions.alwaysTrue())) {
+ return true;
+ }
+
+ PartitionSpec spec = table().specs().get(file.specId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // if the spec is unknown or the table is unpartitioned, be conservative and keep the file
+ return true;
+ }
+
+ try {
+ Evaluator evaluator =
+ evaluatorCache.computeIfAbsent(
+ file.specId(),
+ ignored -> {
+ // an inclusive projection turns predicates on source columns into predicates on
+ // partition values, so non-identity transforms are handled correctly
+ Expression partitionFilter =
+ Projections.inclusive(spec, isCaseSensitive()).project(currentFilter);
+ return new Evaluator(spec.partitionType(), partitionFilter, isCaseSensitive());
+ });
+ return evaluator.eval(file.partition());
+ } catch (Exception e) {
+ // if the filter cannot be projected or evaluated, be conservative and keep the file
+ return true;
+ }
+ }
+
+ /**
+ * Returns the partitions of the added delete files, or an empty set when some delete has global
+ * reach (an unpartitioned spec) and data manifests cannot be pruned by partition.
+ */
+ private PartitionSet addedDeletePartitions(DeleteFileIndex addedDeleteIndex) {
+ PartitionSet partitions = PartitionSet.create(table().specs());
+ for (DeleteFile file : addedDeleteIndex.referencedDeleteFiles()) {
+ PartitionSpec spec = table().specs().get(file.specId());
+ if (spec == null || spec.isUnpartitioned()) {
+ // an empty set disables partition pruning
+ return PartitionSet.create(table().specs());
+ }
+
+ partitions.add(file.specId(), file.partition());
+ }
+
+ return partitions;
+ }
+
+ /**
+ * Returns the data file locations referenced by the added delete files when every added delete is
+ * file-scoped (a DV or a single-file position delete), or null when some added delete may apply
+ * to files other than the ones it references.
+ */
+ private static Set<String> fileScopedTargets(DeleteFileIndex addedDeleteIndex) {
+ Set<String> locations = Sets.newHashSet();
+ for (DeleteFile file : addedDeleteIndex.referencedDeleteFiles()) {
+ String referencedLocation = ContentFileUtil.referencedDataFileLocation(file);
+ if (referencedLocation == null) {
+ return null;
+ }
+
+ locations.add(referencedLocation);
+ }
+
+ return locations;
+ }
+
+ /** Data file changes observed across the changelog snapshots. */
+ private static class ChangedDataFiles {
+ private final Map<Long, Map<String, ManifestEntry.Status>> statusBySnapshot;
+ private final Set<String> deletedFileLocations;
+ private final PartitionSet deletedFilePartitions;
+
+ ChangedDataFiles(
+ Map<Long, Map<String, ManifestEntry.Status>> statusBySnapshot,
+ Set<String> deletedFileLocations,
+ PartitionSet deletedFilePartitions) {
+ this.statusBySnapshot = statusBySnapshot;
+ this.deletedFileLocations = deletedFileLocations;
+ this.deletedFilePartitions = deletedFilePartitions;
+ }
+
+ static ChangedDataFiles empty(Map<Integer, PartitionSpec> specsById) {
+ return new ChangedDataFiles(Map.of(), Set.of(), PartitionSet.create(specsById));
+ }
+ }
+
+ /** Returns true if any snapshot in the range added or removed a delete file. */
+ private static boolean hasDeleteFileChanges(DeleteFileChanges deleteFileChanges) {
+ for (DeleteFileIndex addedDeletes : deleteFileChanges.addedBySnapshot.values()) {
+ if (!addedDeletes.isEmpty()) {
+ return true;
+ }
+ }
+
+ for (List<DeleteFile> removedDeletes : deleteFileChanges.removedBySnapshot.values()) {
+ if (!removedDeletes.isEmpty()) {
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ /** Returns true if the snapshot the range starts from has any delete files. */
+ private boolean hasExistingDeletes(Long fromSnapshotIdExclusive) {
+ if (fromSnapshotIdExclusive == null) {
+ return false;
+ }
+
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ return fromSnapshot != null && !fromSnapshot.deleteManifests(table().io()).isEmpty();
+ }
+
+ private boolean includeDeleteFiles() {
+ boolean tableValue =
+ PropertyUtil.propertyAsBoolean(
+ table().properties(),
+ TableProperties.CHANGELOG_SCAN_INCLUDE_DELETE_FILES,
+ TableProperties.CHANGELOG_SCAN_INCLUDE_DELETE_FILES_DEFAULT);
+ return PropertyUtil.propertyAsBoolean(
+ options(), TableProperties.CHANGELOG_SCAN_INCLUDE_DELETE_FILES, tableValue);
+ }
+
+ /**
+ * Collects, for each snapshot, the statuses of data files changed in it, along with the locations
+ * and partitions of data files removed anywhere in the scan range.
+ */
+ private ChangedDataFiles collectChangedDataFiles(Deque<Snapshot> changelogSnapshots) {
+
+ Map<Long, Map<String, ManifestEntry.Status>> statusBySnapshot = Maps.newConcurrentMap();
+ Queue<Set<String>> localDeletedLocations = new ConcurrentLinkedQueue<>();
+ Queue<PartitionSet> localDeletedPartitions = new ConcurrentLinkedQueue<>();
+
+ Tasks.foreach(changelogSnapshots)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .executeWith(planExecutor())
+ .run(
+ snapshot -> {
+ long snapshotId = snapshot.snapshotId();
+ Map<String, ManifestEntry.Status> fileStatuses = Maps.newHashMap();
+ Set<String> deletedLocations = Sets.newHashSet();
+ PartitionSet deletedPartitions = PartitionSet.create(table().specs());
+
+ List<ManifestFile> changedDataManifests =
+ snapshot.dataManifests(table().io()).stream()
+ .filter(
+ manifest ->
+ manifest.snapshotId() != null && manifest.snapshotId() == snapshotId)
+ .toList();
+
+ if (!changedDataManifests.isEmpty()) {
+ ManifestGroup changedGroup =
+ new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
+ .specsById(table().specs())
+ .caseSensitive(isCaseSensitive())
+ .select(scanColumns())
+ .filterData(filter())
+ .ignoreExisting()
+ .columnsToKeepStats(columnsToKeepStats());
+
+ try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
+ for (ManifestEntry<DataFile> entry : entries) {
+ if (entry.snapshotId() != null && entry.snapshotId() == snapshotId) {
+ fileStatuses.put(entry.file().location(), entry.status());
+ if (entry.status() == ManifestEntry.Status.DELETED) {
+ // copy the file: the reader reuses the entry, so the partition struct
+ // must not be stored as a live reference
+ DataFile deletedFile = entry.file().copyWithoutStats();
+ deletedLocations.add(deletedFile.location());
+ deletedPartitions.add(deletedFile.specId(), deletedFile.partition());
+ }
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to collect file statuses for snapshot " + snapshot.snapshotId(), e);
+ }
+ }
+
+ statusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
+ localDeletedLocations.add(deletedLocations);
+ localDeletedPartitions.add(deletedPartitions);
+ });
+
+ Set<String> deletedFileLocations = Sets.newHashSet();
+ localDeletedLocations.forEach(deletedFileLocations::addAll);
+
+ PartitionSet deletedFilePartitions = PartitionSet.create(table().specs());
+ localDeletedPartitions.forEach(deletedFilePartitions::addAll);
+
+ return new ChangedDataFiles(statusBySnapshot, deletedFileLocations, deletedFilePartitions);
+ }
+
+ /**
+ * Scope of existing (pre-range) delete files that can apply to changelog tasks in the scan range.
+ * Existing deletes are only attached to data files that were removed in the range or hit by newly
+ * added deletes, so delete files outside this scope can be skipped while building the existing
+ * delete index:
+ *
+ * <ul>
+ * <li>file-scoped position deletes and DVs are kept only if they reference an affected data
+ * file location, or lie in a partition where a partition-scoped delete was added
+ * <li>equality deletes and partition-scoped position deletes are kept only if they have global
+ * reach or lie in an affected partition
+ * </ul>
+ */
+ private static class ExistingDeleteScope {
+ private final Map<Integer, PartitionSpec> specsById;
+ private final PartitionSet allAffectedPartitions;
+ private final PartitionSet broadDeletePartitions;
+ private final Set<String> affectedDataFileLocations;
+
+ ExistingDeleteScope(
+ Map<Integer, PartitionSpec> specsById,
+ PartitionSet allAffectedPartitions,
+ PartitionSet broadDeletePartitions,
+ Set<String> affectedDataFileLocations) {
+ this.specsById = specsById;
+ this.allAffectedPartitions = allAffectedPartitions;
+ this.broadDeletePartitions = broadDeletePartitions;
+ this.affectedDataFileLocations = affectedDataFileLocations;
+ }
+
+ boolean keeps(DeleteFile file) {
+ String referencedLocation = ContentFileUtil.referencedDataFileLocation(file);
+ if (referencedLocation != null) {
+ // file-scoped position delete or DV
+ return affectedDataFileLocations.contains(referencedLocation)
+ || broadDeletePartitions.contains(file.specId(), file.partition());
+ }
+
+ // equality delete or partition-scoped position delete
+ PartitionSpec spec = specsById.get(file.specId());
+ return spec == null
+ || spec.isUnpartitioned()
+ || allAffectedPartitions.contains(file.specId(), file.partition());
+ }
+ }
+
+ /**
+ * Derives the scope of existing deletes that can apply to tasks in the scan range from the
+ * removed data files and the added delete files. Returns null when scoping is not possible
+ * because some added delete has global reach and can affect any data file.
+ */
+ private ExistingDeleteScope buildExistingDeleteScope(
+ ChangedDataFiles changedDataFiles, Map<Long, DeleteFileIndex> addedDeletesBySnapshot) {
+ PartitionSet allAffected = PartitionSet.create(table().specs());
+ PartitionSet broadDeletePartitions = PartitionSet.create(table().specs());
+ Set<String> affectedLocations = Sets.newHashSet(changedDataFiles.deletedFileLocations);
+
+ allAffected.addAll(changedDataFiles.deletedFilePartitions);
+
+ for (DeleteFileIndex index : addedDeletesBySnapshot.values()) {
+ for (DeleteFile file : index.referencedDeleteFiles()) {
+ PartitionSpec spec = table().specs().get(file.specId());
+ String referencedLocation = ContentFileUtil.referencedDataFileLocation(file);
+ if (referencedLocation != null) {
+ affectedLocations.add(referencedLocation);
+ } else if (spec == null || spec.isUnpartitioned()) {
+ // an equality delete or a partition-scoped position delete with global reach can
+ // affect any data file, so existing deletes cannot be scoped
+ return null;
+ } else {
+ broadDeletePartitions.add(file.specId(), file.partition());
+ }
+
+ allAffected.add(file.specId(), file.partition());
+ }
+ }
+
+ return new ExistingDeleteScope(
+ table().specs(), allAffected, broadDeletePartitions, affectedLocations);
+ }
+
+ /**
+ * Prunes manifests that cannot contain files in any affected partition. A partitioned manifest
+ * whose spec has no affected partitions is pruned: every delete-to-data match requires equal spec
+ * IDs (file-scoped deletes share their target data file's spec and partition, and
+ * partition-scoped lookups are keyed by the data file's spec), so neither an affected delete file
+ * nor a data file matched by this range's deletes can live in a manifest of an unaffected spec.
+ * Deletes with global reach disable this pruning upstream (null scope for existing deletes; an
+ * empty partition set for per-snapshot data manifest pruning).
+ */
+ private List<ManifestFile> pruneManifestsByAffectedPartitions(
+ List<ManifestFile> manifests, PartitionSet affectedPartitions) {
+ if (affectedPartitions.isEmpty()) {
+ return manifests;
+ }
+
+ Map<Integer, Expression> partitionExprsBySpec =
+ buildAffectedPartitionExpressions(affectedPartitions);
+ if (partitionExprsBySpec == null) {
+ // some affected partition cannot be expressed; skip pruning
+ return manifests;
+ }
+
+ try {
+ Map<Integer, ManifestEvaluator> evaluatorsBySpec = Maps.newHashMap();
+ List<ManifestFile> pruned = Lists.newArrayList();
+ for (ManifestFile manifest : manifests) {
+ PartitionSpec spec = table().specs().get(manifest.partitionSpecId());
+ if (spec == null || spec.isUnpartitioned()) {
+ pruned.add(manifest);
+ continue;
+ }
+
+ Expression expr = partitionExprsBySpec.get(manifest.partitionSpecId());
+ if (expr == null) {
+ // no affected partitions in this manifest's spec: see method javadoc
+ continue;
+ }
+
+ ManifestEvaluator evaluator = evaluatorsBySpec.get(manifest.partitionSpecId());
+ if (evaluator == null) {
+ evaluator = ManifestEvaluator.forPartitionFilter(expr, spec, isCaseSensitive());
+ evaluatorsBySpec.put(manifest.partitionSpecId(), evaluator);
+ }
+
+ if (evaluator.eval(manifest)) {
+ pruned.add(manifest);
+ }
+ }
+
+ return pruned;
+ } catch (Exception e) {
+ // if the partition expressions cannot be evaluated, be conservative and keep all manifests
+ LOG.warn("Failed to prune manifests by affected partitions, skipping pruning", e);
+ return manifests;
+ }
+ }
+
+ /**
+ * Builds partition-space expressions (one per spec) matching the affected partition tuples. The
+ * expressions reference partition field names and compare transformed partition values directly,
+ * so they are correct for non-identity transforms such as bucket or truncate. Returns null if
+ * some affected partition cannot be translated to an expression and pruning must be skipped.
+ */
+ private Map<Integer, Expression> buildAffectedPartitionExpressions(
+ PartitionSet affectedPartitions) {
+ Map<Integer, Expression> exprsBySpec = Maps.newHashMap();
+
+ for (Pair<Integer, StructLike> pair : affectedPartitions) {
+ int specId = pair.first();
+ PartitionSpec spec = table().specs().get(specId);
+ if (spec == null || spec.isUnpartitioned()) {
+ // a partition tuple without a known partitioned spec may affect any manifest
+ return null;
+ }
+
+ StructLike partition = pair.second();
+ Expression tupleExpr = Expressions.alwaysTrue();
+ for (int pos = 0; pos < spec.fields().size(); pos++) {
+ String name = spec.partitionType().fields().get(pos).name();
+ Object value = partition.get(pos, Object.class);
+ if (value == null) {
+ tupleExpr = Expressions.and(tupleExpr, Expressions.isNull(name));
+ } else if (isNaN(value)) {
+ // NaN cannot be expressed as an equality predicate
+ return null;
+ } else {
+ tupleExpr = Expressions.and(tupleExpr, Expressions.equal(name, value));
+ }
+ }
+
+ exprsBySpec.merge(specId, tupleExpr, Expressions::or);
+ }
+
+ return exprsBySpec;
+ }
+
+ private static boolean isNaN(Object value) {
+ return (value instanceof Double && ((Double) value).isNaN())
+ || (value instanceof Float && ((Float) value).isNaN());
+ }
+
+ /**
+ * Tracks the delete files that applied before each snapshot in the scan range and lazily builds a
+ * per-snapshot index of them.
+ *
+ * <p>The index for a snapshot combines deletes that existed before the scan range with deletes
+ * added by earlier snapshots in the range. Delete files that were removed by earlier snapshots in
+ * the range are excluded, so removed delete files are never assigned to tasks.
+ *
+ * <p>Snapshots are consumed oldest to newest, so this class keeps a single advancing cursor over
+ * the range instead of materializing per-snapshot state: one running list of accumulated added
+ * deletes, one running set of removed delete paths, and only the most recently built index.
+ * Within one snapshot, planning worker threads may request the same index concurrently; those
+ * calls are serialized by this object's monitor.
+ */
+ private class CumulativeDeleteIndexes {
+ private final Supplier<DeleteFileIndex> existingDeleteIndexSupplier;
+ private final List<Long> orderedSnapshotIds;
+ private final Map<Long, DeleteFileIndex> addedDeletesBySnapshot;
+ private final Map<Long, List<DeleteFile>> removedDeletesBySnapshot;
+
+ // cursor state: position in orderedSnapshotIds of the next snapshot whose effects to apply,
+ // together with the state produced by all snapshots before that position
+ private int cursor = 0;
+ private final List<DeleteFile> accumulatedAddedDeletes = Lists.newArrayList();
+ private final Set<String> removedPaths = Sets.newHashSet();
+ private Long latestIndexSnapshotId = null;
+ private DeleteFileIndex latestIndex = null;
+
+ CumulativeDeleteIndexes(
+ Deque<Snapshot> snapshots,
+ Map<Long, DeleteFileIndex> addedDeletesBySnapshot,
+ Map<Long, List<DeleteFile>> removedDeletesBySnapshot,
+ Supplier<DeleteFileIndex> existingDeleteIndexSupplier) {
+ this.existingDeleteIndexSupplier = existingDeleteIndexSupplier;
+ this.orderedSnapshotIds = snapshots.stream().map(Snapshot::snapshotId).toList();
+ this.addedDeletesBySnapshot = addedDeletesBySnapshot;
+ this.removedDeletesBySnapshot = removedDeletesBySnapshot;
+ }
+
+ /** Returns an index of all delete files that applied before the given snapshot. */
+ synchronized DeleteFileIndex deletesBefore(long snapshotId) {
+ if (latestIndexSnapshotId != null && latestIndexSnapshotId == snapshotId) {
+ return latestIndex;
+ }
+
+ int position = orderedSnapshotIds.indexOf(snapshotId);
+ Preconditions.checkArgument(
+ position >= 0, "Cannot find snapshot %s in the changelog range", snapshotId);
+
+ // a consumer that re-iterates the plan may ask for a snapshot the cursor already passed;
+ // restarting from the beginning is correct, just slower
+ if (position < cursor) {
+ resetCursor();
+ }
+
+ // apply the effects of every snapshot strictly before the requested one
+ while (cursor < position) {
+ advanceCursor(orderedSnapshotIds.get(cursor));
+ cursor += 1;
+ }
+
+ this.latestIndex = buildIndex();
+ this.latestIndexSnapshotId = snapshotId;
+ return latestIndex;
+ }
+
+ private void resetCursor() {
+ this.cursor = 0;
+ accumulatedAddedDeletes.clear();
+ removedPaths.clear();
+ }
+
+ /** Applies one snapshot's delete file removals and additions to the running state. */
+ private void advanceCursor(long snapshotId) {
+ List<DeleteFile> removedDeletes =
+ removedDeletesBySnapshot.getOrDefault(snapshotId, List.of());
+ if (!removedDeletes.isEmpty()) {
+ Set<String> currentRemovedPaths = Sets.newHashSet();
+ for (DeleteFile removed : removedDeletes) {
+ currentRemovedPaths.add(removed.location());
+ }
+
+ // the cumulative set is still needed to filter pre-range existing delete files, but only
+ // this snapshot's removals may evict accumulated in-range deletes: a path removed earlier
+ // and re-added later in the range is live again
+ removedPaths.addAll(currentRemovedPaths);
+ accumulatedAddedDeletes.removeIf(file -> currentRemovedPaths.contains(file.location()));
+ }
+
+ DeleteFileIndex addedDeleteIndex = addedDeletesBySnapshot.get(snapshotId);
+ if (addedDeleteIndex != null && !addedDeleteIndex.isEmpty()) {
+ for (DeleteFile file : addedDeleteIndex.referencedDeleteFiles()) {
+ accumulatedAddedDeletes.add(file);
+ }
+ }
+ }
+
+ private DeleteFileIndex buildIndex() {
+ List<DeleteFile> candidates = Lists.newArrayList();
+ for (DeleteFile file : existingDeleteIndexSupplier.get().referencedDeleteFiles()) {
+ if (!removedPaths.contains(file.location())) {
+ candidates.add(file);
+ }
+ }
+
+ candidates.addAll(accumulatedAddedDeletes);
+
+ if (candidates.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // A data file has at most one live DV, but when a DV was replaced by a snapshot this scan
+ // does not track (e.g. a mid-range REPLACE rewrote it), both versions may be accumulated.
+ // Keep only the newest DV per referenced data file: DVs are cumulative, so the newest one
+ // carries all previously deleted positions. Candidates are ordered oldest to newest.
+ //
+ // When a skipped REPLACE snapshot rewrote a delete file, the DV retained here is the
+ // pre-range one, which is content-equivalent to the rewritten one but may already have been
Review Comment:
DV could change after rewritten when data files are changed as well.
```
A = [a, b, c], DV_A = {1} -> [a, c]
C = [b, a, c], DV_C = {0} -> [a, c]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org