(iceberg) branch main updated: Core: Skip unnecessary manifest scans during expire snapshots metadata cleanup (#16691)

[hu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

huaxingao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git


The following commit(s) were added to refs/heads/main by this push:
     new 13182abb18 Core: Skip unnecessary manifest scans during expire snapshots metadata cleanup (#16691)
13182abb18 is described below

commit 13182abb185220787d2ce13dd9ea025905f83800
Author: Yujiang Zhong <42...@users.noreply.github.com>
AuthorDate: Sun Jun 7 09:14:39 2026 +0800

    Core: Skip unnecessary manifest scans during expire snapshots metadata cleanup (#16691)
    
    * Core: Skip manifest scan in expire snapshots when only the current spec exists
    
    When cleanExpiredMetadata is enabled, avoid scanning all snapshot manifest
    lists to collect reachable partition spec IDs if the table has only a single
    spec that is the current default spec. This eliminates unnecessary I/O for
    the common single-spec table case.
    
    Co-Authored-By: Claude Opus 4.6 (1M context) <no...@anthropic.com>
    
    * Core: Skip manifest scan once all specs are confirmed reachable
    
    When collecting reachable partition specs during expire snapshots, skip
    scanning remaining snapshot manifests once all known specs have been found.
    This avoids unnecessary I/O for tables with few spec evolutions.
    
    Co-Authored-By: Claude Opus 4.6 (1M context) <no...@anthropic.com>
    
    * Core: Simplify mayHaveExpiredSpecs check to specs size only
    
    The default spec is always present in the specs list, so when
    specs().size() == 1 the sole spec is guaranteed to be the default.
    The additional anyMatch check was unreachable.
    
    Co-Authored-By: Claude Opus 4.6 (1M context) <no...@anthropic.com>
    
    ---------
    
    Co-authored-by: zhongyujiang <zh...@xiaomi.com>
    Co-authored-by: Claude Opus 4.6 (1M context) <no...@anthropic.com>
---
 core/src/main/java/org/apache/iceberg/RemoveSnapshots.java | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java b/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java
index 8e320b3d69..dbccdb86ed 100644
--- a/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java
+++ b/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java
@@ -242,14 +242,18 @@ class RemoveSnapshots implements ExpireSnapshots {
       Set<Integer> reachableSchemas = Sets.newConcurrentHashSet();
       reachableSchemas.add(base.currentSchemaId());
 
+      boolean mayHaveExpiredSpecs = base.specs().size() > 1;
+
       Tasks.foreach(idsToRetain)
           .executeWith(planExecutorService())
           .run(
               snapshotId -> {
                 Snapshot snapshot = base.snapshot(snapshotId);
-                snapshot.allManifests(ops.io()).stream()
-                    .map(ManifestFile::partitionSpecId)
-                    .forEach(reachableSpecs::add);
+                if (mayHaveExpiredSpecs && reachableSpecs.size() < base.specs().size()) {
+                  snapshot.allManifests(ops.io()).stream()
+                      .map(ManifestFile::partitionSpecId)
+                      .forEach(reachableSpecs::add);
+                }
                 reachableSchemas.add(snapshot.schemaId());
               });