You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "rinzool (via GitHub)" <gi...@apache.org> on 2026/08/25 15:14:25 UTC

[PR] fix: propagate SupportsBulkOperations through the FileIO wrappers [polaris]

rinzool opened a new pull request, #5378:
URL: https://github.com/apache/polaris/pull/5378

   ## Background
   
   #4850 added batched deletion to the file cleanup tasks: `BatchFileCleanupTaskHandler` hands the
   whole batch to `FileCleanupTaskHandler.tryDelete(..., Iterable<String> files, type, isConcurrent,
   ...)`, which calls `CatalogUtil.deleteFiles`.
   
   `CatalogUtil.deleteFiles` batches only when the FileIO satisfies
   `instanceof SupportsBulkOperations`, and falls back to one `deleteFile` per path otherwise.
   
   The FileIO that reaches the cleanup tasks is never the configured one. It is wrapped by
   `DefaultFileIOFactory` in `ExceptionMappingFileIO`, and on Azure wrapped again by
   `WasbTranslatingFileIOFactory` in `WasbTranslatingFileIO`. Both declared only `FileIO`, and a
   capability held by the wrapped object is not visible through the wrapper: `instanceof` tests the
   wrapper's own type. The check was therefore always false and the per-file branch always ran, so the
   batching added by #4850 never executed.
   
   This affects every storage backend, since each FileIO Polaris selects by storage type implements
   `DelegateFileIO`, which extends `SupportsBulkOperations`: `S3FileIO`, `GCSFileIO`, `ADLSFileIO` and
   `HadoopFileIO`.
   
   Fixes #5377
   
   ## Change
   
   Both wrappers gain a `wrap(FileIO)` factory that returns a subclass implementing
   `SupportsBulkOperations`, delegating `deleteFiles` to the wrapped FileIO — but only when the wrapped
   FileIO actually supports bulk operations. The constructors become `protected` so callers go through
   the factory and cannot silently reintroduce the problem. Both factories now call `wrap()`.
   
   `WasbTranslatingFileIO` translates each path on the bulk path, as it already does for `deleteFile`.
   It passes a re-iterable `Iterable` rather than a one-shot iterator, since an implementation may
   traverse the paths more than once (for example to size the batch before deleting).
   
   Advertising the capability unconditionally was deliberately avoided: it would send callers down a
   bulk path the wrapped FileIO cannot serve. Implementing `DelegateFileIO` was also avoided, because
   it would additionally claim `SupportsPrefixOperations`, which these wrappers do not delegate.
   
   ## Effect
   
   Measured on a downstream deployment carrying the same wrappers, dropping a table with 660 data
   files across 66 manifests (859 files deleted in total):
   
   | | requests to object storage |
   |---|---|
   | wrappers as they were | 859 `DeleteObject` + 859 `HeadObject` |
   | with the capability propagated | 859 files deleted in 86 batched calls |
   
   On S3 each delete is a billable request, so this reduces cost as well as latency.
   
   ## Tests
   
   `ExceptionMappingFileIOTest`:
   - the capability is advertised when the wrapped FileIO supports bulk operations, and not otherwise
   - `deleteFiles` delegates once to the wrapped FileIO and issues no per-file `deleteFile`
   - an `UnknownHostException` raised by the wrapped `deleteFiles` is still remapped to
     `FileIOUnknownHostException`
   
   `WasbTranslatingFileIOTest` (new — the class had no test coverage):
   - the same capability assertions
   - every path is translated from WASB to ABFS on the bulk path, and non-Azure paths pass through
   - the `Iterable` handed to the delegate can be iterated more than once
   
   The existing tests in `ExceptionMappingFileIOTest` were switched to the `wrap()` factory and to
   AssertJ, matching the assertion style used elsewhere in the repo.
   
   Verified with `./gradlew format compileAll` and `./gradlew check` on Java 21.
   
   ## Notes
   
   Two follow-ups were left out to keep this change focused, and can be filed separately if useful:
   
   - `ManifestFileCleanupTaskHandler` still deletes manifest data files one per file; it never adopted
     the batch method from #4850.
   - `TableCleanupTaskHandler` builds each cleanup task's name by concatenating the batch's file list.
     Since the name is part of a unique index, raising `TABLE_METADATA_CLEANUP_BATCH_SIZE` makes the
     insert exceed the PostgreSQL btree limit, the parent cleanup task fails, and the dropped table's
     files are never removed. This currently caps metadata batches at roughly 26 paths.
   
   AI assistance was used in preparing this change (investigation, drafting and tests); the
   implementation and its rationale have been reviewed and are my responsibility.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: propagate SupportsBulkOperations through the FileIO wrappers [polaris]

Posted by "nandorKollar (via GitHub)" <gi...@apache.org>.
nandorKollar commented on PR #5378:
URL: https://github.com/apache/polaris/pull/5378#issuecomment-5560436973

   @rinzool could you please resolve the merge conflict on the changelog?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: propagate SupportsBulkOperations through the FileIO wrappers [polaris]

Posted by "rinzool (via GitHub)" <gi...@apache.org>.
rinzool commented on PR #5378:
URL: https://github.com/apache/polaris/pull/5378#issuecomment-5450867746

   > LGTM.. Note that `BatchFileCleanupTaskHandler` still does a HEAD per file via `TaskUtils.exists` before delete. This pr batches the deletes, but the exists check remains... Worth a follow-up?
   > 
   > Thanks for contribution @rinzool
   
   Hi @vigneshio 
   Good catch
   I checked all provider, they should not raise any error in case of missing file when calling the bulk delete. Therefore we could totally drop the check of existing files on classes that support `SupportsBulkOperations` and therefore skip those head object.
   
   I opened a Draft PR on top on this one to see what we could do. If you want, I can include that in this PR or open a PR after this one is merged for this fix.
   https://github.com/rinzool/polaris/pull/2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: propagate SupportsBulkOperations through the FileIO wrappers [polaris]

Posted by "nandorKollar (via GitHub)" <gi...@apache.org>.
nandorKollar commented on PR #5378:
URL: https://github.com/apache/polaris/pull/5378#issuecomment-5528701489

   Thanks for looking into this. The fix looks good to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: propagate SupportsBulkOperations through the FileIO wrappers [polaris]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #5378:
URL: https://github.com/apache/polaris/pull/5378#discussion_r3930958978


##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/io/WasbTranslatingFileIO.java:
##########
@@ -91,4 +104,24 @@ public void initialize(Map<String, String> properties) {
   public void close() {
     io.close();
   }
+
+  private static class BulkWasbTranslatingFileIO extends WasbTranslatingFileIO
+      implements SupportsBulkOperations {
+
+    private final SupportsBulkOperations bulkIo;
+
+    private BulkWasbTranslatingFileIO(FileIO io) {
+      super(io);
+      this.bulkIo = (SupportsBulkOperations) io;
+    }
+
+    @Override
+    public void deleteFiles(Iterable<String> paths) throws BulkDeletionFailureException {
+      bulkIo.deleteFiles(
+          () ->
+              StreamSupport.stream(paths.spliterator(), false)
+                  .map(WasbTranslatingFileIO::translate)
+                  .iterator());
+    }

Review Comment:
   maybe we could do this in one line
   ```
   bulkIo.deleteFiles(Iterables.transform(paths, WasbTranslatingFileIO::translate));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: propagate SupportsBulkOperations through the FileIO wrappers [polaris]

Posted by "rinzool (via GitHub)" <gi...@apache.org>.
rinzool commented on PR #5378:
URL: https://github.com/apache/polaris/pull/5378#issuecomment-5586915137

   > @rinzool could you please resolve the merge conflict on the changelog?
   
   @nandorKollar  Conflict is resolved :+1: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: propagate SupportsBulkOperations through the FileIO wrappers [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #5378:
URL: https://github.com/apache/polaris/pull/5378#issuecomment-5636108879

   Nice catch, @rinzool ! Thanks for the fix!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: propagate SupportsBulkOperations through the FileIO wrappers [polaris]

Posted by "rinzool (via GitHub)" <gi...@apache.org>.
rinzool commented on code in PR #5378:
URL: https://github.com/apache/polaris/pull/5378#discussion_r3935353786


##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/io/WasbTranslatingFileIO.java:
##########
@@ -91,4 +104,24 @@ public void initialize(Map<String, String> properties) {
   public void close() {
     io.close();
   }
+
+  private static class BulkWasbTranslatingFileIO extends WasbTranslatingFileIO
+      implements SupportsBulkOperations {
+
+    private final SupportsBulkOperations bulkIo;
+
+    private BulkWasbTranslatingFileIO(FileIO io) {
+      super(io);
+      this.bulkIo = (SupportsBulkOperations) io;
+    }
+
+    @Override
+    public void deleteFiles(Iterable<String> paths) throws BulkDeletionFailureException {
+      bulkIo.deleteFiles(
+          () ->
+              StreamSupport.stream(paths.spliterator(), false)
+                  .map(WasbTranslatingFileIO::translate)
+                  .iterator());
+    }

Review Comment:
   Good call, thanks! I updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: propagate SupportsBulkOperations through the FileIO wrappers [polaris]

Posted by "nandorKollar (via GitHub)" <gi...@apache.org>.
nandorKollar commented on PR #5378:
URL: https://github.com/apache/polaris/pull/5378#issuecomment-5632750612

   Merged, thanks @rinzool for fixing this and @flyingImer @ayushtkn @vigneshio for the reviews!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: propagate SupportsBulkOperations through the FileIO wrappers [polaris]

Posted by "nandorKollar (via GitHub)" <gi...@apache.org>.
nandorKollar merged PR #5378:
URL: https://github.com/apache/polaris/pull/5378


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org