You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kinolaev (via GitHub)" <gi...@apache.org> on 2026/03/21 16:27:55 UTC

[PR] Core: close entries iterable in ManifestFilterManager [iceberg]

kinolaev opened a new pull request, #15713:
URL: https://github.com/apache/iceberg/pull/15713

   This PR prevents deadlocks while filtering manifest entries. The problem is `ManifestFilterManager.filterManifest` in some cases reads each manifest twice: first in `manifestHasDeletedFiles` and then in `filterManifestWithDeletedFiles`. If `manifestHasDeletedFiles` returns in the middle of entries iterable, the underlying connection is open until the ManifestReader is closed. `filterManifest` method is called for all manifests in parallel. When number of simultaneous connections is limited (for example by http-client.apache.max-connections) it can lead to a deadlock because all connections are held by `manifestHasDeletedFiles`.
   
   The problem can be reproduced using spark-sql with S3FileIO and `http-client.apache.max-connections=1`:
   ```sql
   create table manifestfiltermanager(id bigint)
     partitioned by (truncate(1, id))
     tblproperties('write.delete.mode'='merge-on-read');
   -- create a data manifest with 100 entries
   insert into manifestfiltermanager select id from range(100);
   -- create a delete manifest with 50 entries
   delete from manifestfiltermanager where id in (select id from range(0, 100, 2));
   -- make delete files dangling (fails without https://github.com/apache/iceberg/pull/15712)
   call system.rewrite_data_files('manifestfiltermanager', options => map('rewrite-all', 'true'));
   -- ManifestFilterManager.manifestHasDeletedFiles reads first block of the delete manifest
   -- and returns true before the end of liveEntries() iterable without closing it.
   -- ManifestFilterManager.filterManifestWithDeletedFiles fails with ConnectionPoolTimeoutException
   call system.rewrite_data_files('manifestfiltermanager', options => map('rewrite-all', 'true'));
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-4238243434

   Hello @anoopj , is there anything I can do to move the PR forward?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r2989043129


##########
core/src/main/java/org/apache/iceberg/ManifestFilterManager.java:
##########


Review Comment:
   This has the same problem. Perhaps fix this as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Call close on iterables returned by ManifestReader.liveEntries() in ManifestFilterManager [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-5294432381

   Merged to main.
   Thanks @kinolaev for the fix and @nastra and @anoopj for the reviews!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-4891150457

   @kinolaev can you please rebase and fix the merge conflict?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3527528140


##########
core/src/main/java/org/apache/iceberg/ManifestFilterManager.java:
##########
@@ -455,36 +456,40 @@ private boolean manifestHasDeletedFiles(
 
     boolean isDelete = reader.isDeleteManifestReader();
 
-    for (ManifestEntry<F> entry : reader.liveEntries()) {

Review Comment:
   the reader is being opened/closed one level further above in https://github.com/apache/iceberg/blob/c31dd921e1550c643a3ea1fce9d9573cc06e2607/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L380 and then reused across these two methods. Why is it now better to open/close the reader twice instead of once?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3783114969


##########
core/src/test/java/org/apache/iceberg/TestDeleteFiles.java:
##########
@@ -738,7 +744,120 @@ public void removingDataFilesWhenTruncatingAlsoRemovesDVs() {
         statuses(Status.DELETED, Status.DELETED));
   }
 
+  @TestTemplate
+  public void testFilterManifestsWithLimitedIOPool() {
+    TrackingFileIO io = new TrackingFileIO(FILE_IO);
+    TestTables.TestTableOperations ops =
+        new TestTables.TestTableOperations("limited-test", tableDir, io);
+    Table testTable =
+        TestTables.create(
+            tableDir, "limited-test", SCHEMA, SPEC, SortOrder.unsorted(), formatVersion, ops);
+    commit(testTable, testTable.newFastAppend().appendFile(FILE_A), branch);
+    Snapshot deleteSnap = commit(testTable, testTable.newDelete().deleteFile(FILE_A), branch);
+    assertThat(deleteSnap.summary()).containsEntry(SnapshotSummary.DELETED_FILES_PROP, "1");
+    assertThat(io.peakCount()).isEqualTo(1);
+  }
+
   private static ByteBuffer longToBuffer(long value) {
     return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, value);
   }
+
+  /** A {@link FileIO} that tracks input streams. */
+  private static class TrackingFileIO implements FileIO {

Review Comment:
   We could save a few lines of code if we use `Mockito` to spy on `LocalFileIO`.
   Maybe something like this:
   ```
     /** Returns a {@link FileIO} that records the peak number of concurrently open input streams. */
     private static FileIO trackingIO(AtomicInteger peakOpenStreams) {
       AtomicInteger openStreams = new AtomicInteger(0);
       FileIO io = Mockito.spy(new TestTables.LocalFileIO());
   
       Mockito.doAnswer(
               newInputFile -> {
                 InputFile file = Mockito.spy((InputFile) newInputFile.callRealMethod());
                 Mockito.doAnswer(
                         newStream -> {
                           peakOpenStreams.accumulateAndGet(openStreams.incrementAndGet(), Math::max);
                           SeekableInputStream stream =
                               Mockito.spy((SeekableInputStream) newStream.callRealMethod());
                           Mockito.doAnswer(
                                   close -> {
                                     openStreams.decrementAndGet();
                                     return close.callRealMethod();
                                   })
                               .when(stream)
                               .close();
                           return stream;
                         })
                     .when(file)
                     .newStream();
                 return file;
               })
           .when(io)
           .newInputFile(Mockito.anyString());
   
       return io;
     }
   ```
   
   What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-4794853181

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3527770656


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -410,4 +411,106 @@ public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailu
       throw new RuntimeException("Expected to mock this function");
     }
   }
+
+  /** A {@link FileIO} that enforces a limit on concurrent open input streams. */
+  public static class TrackingFileIO implements FileIO {

Review Comment:
   the other alternative to testing this would be to have something like the following, which wouldn't require introducing a separate FileIO class in `TestTables` and we could keep everything isolated in a single test class:
   
   ```
   @ExtendWith(ParameterizedTestExtension.class)
   class TestManifestFilterManager extends TestBase {
     
     @BeforeEach
     public void before() {
       LiveEntriesTracker.reset();
     }
   
     @TestTemplate
     public void filterManifestsClosesLiveEntriesBeforeReopening() throws IOException {
       ExecutorService executor = Executors.newSingleThreadExecutor();
       try {
         ManifestFile manifest = writeManifest(1L, FILE_A);
         TestDataFileFilterManager filterManager = new TestDataFileFilterManager(executor);
         filterManager.delete(FILE_A);
   
         List<ManifestFile> filtered = filterManager.filterManifests(SCHEMA, List.of(manifest));
   
         assertThat(filtered).hasSize(1);
         assertThat(filtered.get(0).path()).isNotEqualTo(manifest.path());
         assertThat(LiveEntriesTracker.peakOpen())
             .as("liveEntries() must be closed before opening again during manifest rewrite")
             .isEqualTo(1);
       } finally {
         executor.shutdownNow();
       }
     }
   
     private static final class LiveEntriesTracker {
       private static final AtomicInteger OPEN = new AtomicInteger(0);
       private static final AtomicInteger PEAK = new AtomicInteger(0);
   
       private LiveEntriesTracker() {}
   
       static void reset() {
         OPEN.set(0);
         PEAK.set(0);
       }
   
       static int peakOpen() {
         return PEAK.get();
       }
   
       static <T> CloseableIterable<T> track(CloseableIterable<T> iterable) {
         PEAK.accumulateAndGet(OPEN.incrementAndGet(), Math::max);
         return new CloseableIterable<T>() {
           @Override
           public void close() throws IOException {
             iterable.close();
             OPEN.decrementAndGet();
           }
   
           @Override
           public CloseableIterator<T> iterator() {
             return iterable.iterator();
           }
         };
       }
     }
   
     private static class TrackingManifestReader extends ManifestReader<DataFile> {
       TrackingManifestReader(
           ManifestFile manifest, FileIO io, Map<Integer, PartitionSpec> specsById) {
         super(
             io.newInputFile(manifest.path()),
             manifest.partitionSpecId(),
             specsById,
             InheritableMetadataFactory.fromManifest(manifest),
             manifest.firstRowId(),
             true,
             FileType.DATA_FILES);
       }
   
       @Override
       CloseableIterable<ManifestEntry<DataFile>> liveEntries() {
         return LiveEntriesTracker.track(super.liveEntries());
       }
     }
   
     private class TestDataFileFilterManager extends ManifestFilterManager<DataFile> {
       private final AtomicInteger manifestCounter = new AtomicInteger(0);
   
       TestDataFileFilterManager(ExecutorService executor) {
         super(table.specs(), (Supplier<ExecutorService>) () -> executor);
       }
   
       @Override
       protected void deleteFile(String location) {
         // filtering test only
       }
   
       @Override
       protected ManifestWriter<DataFile> newManifestWriter(PartitionSpec spec) {
         String path =
             String.format(
                 "%s/filtered-%d%s",
                 table.location(),
                 manifestCounter.incrementAndGet(),
                 manifestFormat().addExtension(""));
         OutputFile outputFile = FILE_IO.newOutputFile(path);
         return ManifestFiles.write(formatVersion, spec, outputFile, 1L);
       }
   
       @Override
       protected ManifestReader<DataFile> newManifestReader(ManifestFile manifest) {
         return new TrackingManifestReader(manifest, FILE_IO, table.specs());
       }
   
       @Override
       protected Set<DataFile> newFileSet() {
         return DataFileSet.create();
       }
     }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r2989228952


##########
core/src/main/java/org/apache/iceberg/ManifestFilterManager.java:
##########


Review Comment:
   `filterManifestWithDeletedFiles` is called only in `filterManifest`, where `reader` is already wrapped with try-with-resources. So it doesn't cause any issue. But if you'd like I can wrap this liveEntries() call too, just in case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3527671277


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -408,4 +411,106 @@ public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailu
       throw new RuntimeException("Expected to mock this function");
     }
   }
+
+  /** A {@link FileIO} that enforces a limit on concurrent open input streams. */
+  public static class TrackingFileIO implements FileIO {
+
+    private final FileIO delegate;
+    private final AtomicInteger count = new AtomicInteger(0);
+    private final AtomicInteger peakCount = new AtomicInteger(0);
+
+    TrackingFileIO(FileIO delegate) {
+      this.delegate = delegate;
+    }
+
+    @Override
+    public InputFile newInputFile(String path) {
+      return new TrackingInputFile(delegate.newInputFile(path), count, peakCount);
+    }
+
+    @Override
+    public OutputFile newOutputFile(String path) {
+      return delegate.newOutputFile(path);
+    }
+
+    @Override
+    public void deleteFile(String path) {
+      delegate.deleteFile(path);
+    }
+
+    public int getPeakCount() {

Review Comment:
   Renamed, thanks @nastra



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3782876314


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -410,4 +411,106 @@ public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailu
       throw new RuntimeException("Expected to mock this function");
     }
   }
+
+  /** A {@link FileIO} that enforces a limit on concurrent open input streams. */
+  public static class TrackingFileIO implements FileIO {

Review Comment:
   Thanks, @pvary, I tried extending `TrackingFileIO` to reuse it in `TestSparkExecutorCache` and `TestStreamingCheckpointHadoopIO`. However, while they look similar, they are initialized differently and track different things. There isn't much to reuse, so it's not worth it. I moved `TrackingFileIO` next to the test that uses it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-4530721088

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3774523417


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -410,4 +411,106 @@ public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailu
       throw new RuntimeException("Expected to mock this function");
     }
   }
+
+  /** A {@link FileIO} that enforces a limit on concurrent open input streams. */
+  public static class TrackingFileIO implements FileIO {

Review Comment:
   @pvary, I wasn't planning to reuse `TrackingFileIO` in `TestSparkExecutorCache`, it was just an example to show that it could be a useful test utility. But if you think it's a good idea, I will update `TestSparkExecutorCache` in a follow-up PR after this one is merged (or even before if you want).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3528286604


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -410,4 +411,106 @@ public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailu
       throw new RuntimeException("Expected to mock this function");
     }
   }
+
+  /** A {@link FileIO} that enforces a limit on concurrent open input streams. */
+  public static class TrackingFileIO implements FileIO {

Review Comment:
   Thanks @nastra, that is definitely a good alternative. But do we want to test:
   1) that `ManifestFilterManager` closes `liveEntries()` before opening it again, or
   2) that a single file commit requires only one connection from a connection pool?
   I intentionally chose the second approach to ensure there are no other similar bugs in the commit path. And I think `TrackingFileIO` might be useful for other tests in the future, which is why I put it in `TestTables`.
   
   I ran your test and it works as expected. Let me know if you still prefer to keep things isolated, and I'll go ahead and replace the test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-5178485908

   Hello @pvary , could you please take a look at this PR? Despite the line count, it only wraps `reader.liveEntries()` with try-with-resources in two places, so it shouldn't take much time to review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3527609057


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -408,4 +411,106 @@ public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailu
       throw new RuntimeException("Expected to mock this function");
     }
   }
+
+  /** A {@link FileIO} that enforces a limit on concurrent open input streams. */
+  public static class TrackingFileIO implements FileIO {
+
+    private final FileIO delegate;
+    private final AtomicInteger count = new AtomicInteger(0);
+    private final AtomicInteger peakCount = new AtomicInteger(0);
+
+    TrackingFileIO(FileIO delegate) {
+      this.delegate = delegate;
+    }
+
+    @Override
+    public InputFile newInputFile(String path) {
+      return new TrackingInputFile(delegate.newInputFile(path), count, peakCount);
+    }
+
+    @Override
+    public OutputFile newOutputFile(String path) {
+      return delegate.newOutputFile(path);
+    }
+
+    @Override
+    public void deleteFile(String path) {
+      delegate.deleteFile(path);
+    }
+
+    public int getPeakCount() {

Review Comment:
   ```suggestion
       public int peakCount() {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3528286604


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -410,4 +411,106 @@ public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailu
       throw new RuntimeException("Expected to mock this function");
     }
   }
+
+  /** A {@link FileIO} that enforces a limit on concurrent open input streams. */
+  public static class TrackingFileIO implements FileIO {

Review Comment:
   Thanks @nastra, that is definitely a good alternative. But do we want to test:
   1) that `ManifestFilterManager` closes `liveEntries()` before opening it again, or
   2) that a single file commit requires only one connection from a connection pool?
   
   I intentionally chose the second approach to ensure there are no other similar bugs in the commit path. And I think `TrackingFileIO` might be useful for other tests in the future, which is why I put it in `TestTables`.
   
   I ran your test and it works as expected. Let me know if you still prefer to keep things isolated, and I'll go ahead and replace the test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3774007618


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -410,4 +411,106 @@ public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailu
       throw new RuntimeException("Expected to mock this function");
     }
   }
+
+  /** A {@link FileIO} that enforces a limit on concurrent open input streams. */
+  public static class TrackingFileIO implements FileIO {

Review Comment:
   @kinolaev: Are you planning to actually create a test for that and reuse the `TrackingFileIO`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3783117417


##########
core/src/test/java/org/apache/iceberg/TestDeleteFiles.java:
##########
@@ -738,7 +744,120 @@ public void removingDataFilesWhenTruncatingAlsoRemovesDVs() {
         statuses(Status.DELETED, Status.DELETED));
   }
 
+  @TestTemplate
+  public void testFilterManifestsWithLimitedIOPool() {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3781510111


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -410,4 +411,106 @@ public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailu
       throw new RuntimeException("Expected to mock this function");
     }
   }
+
+  /** A {@link FileIO} that enforces a limit on concurrent open input streams. */
+  public static class TrackingFileIO implements FileIO {

Review Comment:
   Actually, I have asked, because if we don't reuse, then we probably don't want to expose, and keep it local



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3783255054


##########
core/src/test/java/org/apache/iceberg/TestDeleteFiles.java:
##########
@@ -738,7 +744,120 @@ public void removingDataFilesWhenTruncatingAlsoRemovesDVs() {
         statuses(Status.DELETED, Status.DELETED));
   }
 
+  @TestTemplate
+  public void testFilterManifestsWithLimitedIOPool() {
+    TrackingFileIO io = new TrackingFileIO(FILE_IO);
+    TestTables.TestTableOperations ops =
+        new TestTables.TestTableOperations("limited-test", tableDir, io);
+    Table testTable =
+        TestTables.create(
+            tableDir, "limited-test", SCHEMA, SPEC, SortOrder.unsorted(), formatVersion, ops);
+    commit(testTable, testTable.newFastAppend().appendFile(FILE_A), branch);
+    Snapshot deleteSnap = commit(testTable, testTable.newDelete().deleteFile(FILE_A), branch);
+    assertThat(deleteSnap.summary()).containsEntry(SnapshotSummary.DELETED_FILES_PROP, "1");
+    assertThat(io.peakCount()).isEqualTo(1);
+  }
+
   private static ByteBuffer longToBuffer(long value) {
     return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, value);
   }
+
+  /** A {@link FileIO} that tracks input streams. */
+  private static class TrackingFileIO implements FileIO {

Review Comment:
   I think this is how I should have implemented it in the first place! I checked, the updated test catches the problem. Thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Call close on iterables returned by ManifestReader.liveEntries() in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-5294670514

   Thanks @pvary! Here is the backport PR #17654 to the 1.11.x branch. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-4891349032

   The merge conflict is fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3527528140


##########
core/src/main/java/org/apache/iceberg/ManifestFilterManager.java:
##########
@@ -455,36 +456,40 @@ private boolean manifestHasDeletedFiles(
 
     boolean isDelete = reader.isDeleteManifestReader();
 
-    for (ManifestEntry<F> entry : reader.liveEntries()) {

Review Comment:
   the reader is being opened/closed one level further above in https://github.com/apache/iceberg/blob/c31dd921e1550c643a3ea1fce9d9573cc06e2607/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L380 and then reused across these two methods. Why is it now better to open/close the reader twice instead of once?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-4313468407

   Hello @nastra and @amogh-jahagirdar , I noticed that you were the last people to update ManifestFilterManager. Could you please take a look at this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-4533453257

   The PR is not stale


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-4138654814

   Thanks for the review, @anoopj ! I've added a test in 828f98e1a7b5d25986fd120dd953ecf4763550f1.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#issuecomment-4240366872

   @kinolaev I recommend getting this reviewed with a committer. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r2989350228


##########
core/src/main/java/org/apache/iceberg/ManifestFilterManager.java:
##########


Review Comment:
   Fixed, just in case, looks a little safer now)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3577523769


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -410,4 +411,106 @@ public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailu
       throw new RuntimeException("Expected to mock this function");
     }
   }
+
+  /** A {@link FileIO} that enforces a limit on concurrent open input streams. */
+  public static class TrackingFileIO implements FileIO {

Review Comment:
   For example, `TrackingFileIO` could be extended to track the per-path stream count and used in [TestSparkExecutorCache](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/TestSparkExecutorCache.java#L620-L676)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #15713:
URL: https://github.com/apache/iceberg/pull/15713


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: close entries iterable in ManifestFilterManager [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15713:
URL: https://github.com/apache/iceberg/pull/15713#discussion_r3783099886


##########
core/src/test/java/org/apache/iceberg/TestDeleteFiles.java:
##########
@@ -738,7 +744,120 @@ public void removingDataFilesWhenTruncatingAlsoRemovesDVs() {
         statuses(Status.DELETED, Status.DELETED));
   }
 
+  @TestTemplate
+  public void testFilterManifestsWithLimitedIOPool() {

Review Comment:
   remove `test` from the name of the method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org