You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "steveloughran (via GitHub)" <gi...@apache.org> on 2025/01/22 19:48:13 UTC
[I] HadoopFileIO to support bulk delete through the Hadoop Filesystem APIs [iceberg]
steveloughran opened a new issue, #12055:
URL: https://github.com/apache/iceberg/issues/12055
### Feature Request / Improvement
Hadoop Filesystems now support paged bulk delete API.
For most filesystems the page size is 1; it simply mapped to a single file delete.
For S3A, the page size is the value of `fs.s3a.bulk.delete.page.size`
-each page of deletions is executed as a single bulk delete POST in the AWS API
There are no attempts to implement POSIX "safety checks" for the path being a directory,
the parent directory existing afterwards etc.
As such it is the most efficient way to delete many objects, whose performance
should match that of to `S3FileIO.deleteFiles()`.
For filesystems without bulk delete, this is mapped to delete(path), so
no less efficient than normal delete calls.
Support this in Iceberg:
* Add new option `iceberg.hadoop.bulk.delete.enabled` (default: false)
* Use reflection to use the bulk delete API through the reflection friendly
`org.apache.hadoop.io.wrappedio.WrappedIO` class
* Switch to using the bulk delete mechanism if enabled and present
If active, bulk delete is supported by:
1. building up a page of paths to delete for every target filesystem.
2. Initiate an asynchronous bulk delete request whenever a page is full.
3. When the end of the list has been reached: queue page deletes for
all incomplete pages.
4. Await results and report failures as such.
Missing files are not reported as failures -these are not detected.
Failures will be in permissions, network and possibly transient endpoint issues.
Adds a parameterized test to verify bulk delete works.
This needs to be run against hadoop 3.4.1 to actually verify coverage.
Testing this feature all the way to S3 is complicated.
A test within the hadoop-aws module can validate
the feature through HadoopFileIO and act as regression
testing for the S3A Connector.
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HadoopFileIO to support bulk delete through the Hadoop Filesystem APIs [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12055: HadoopFileIO to support bulk delete through the Hadoop Filesystem APIs
URL: https://github.com/apache/iceberg/issues/12055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HadoopFileIO to support bulk delete through the Hadoop Filesystem APIs [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12055:
URL: https://github.com/apache/iceberg/issues/12055#issuecomment-3100059296
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HadoopFileIO to support bulk delete through the Hadoop Filesystem APIs [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12055:
URL: https://github.com/apache/iceberg/issues/12055#issuecomment-3152865295
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org