You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "steveloughran (via GitHub)" <gi...@apache.org> on 2026/02/18 14:20:10 UTC
[I] Optimise HadoopFileIO for cloud IO [iceberg]
steveloughran opened a new issue, #15353:
URL: https://github.com/apache/iceberg/issues/15353
### Feature Request / Improvement
The class HadoopFileIO uses the hadoop 2 APIs for opening and creating files. By moving to the newer createFile/openFile APIs and passing in information, it should be possible to
* Eliminate head requests when opening files (s3a, abfs)
* Switch the cloud connector to an optimal read policy for the known file type (random, sequential,..) (s3a, gcs)
* Have the s3a connector skip checks that creating a file won't overwrite a directory. This is consistent with S3OutputFile and relies on iceberg creating unique filenames, or at least not creating files above other objects.
There are some other minor tunings.
Constraints
1. All of this shall use APIs in Hadoop 3.3.5 so there's no need to use reflection for spark 3.4+ use.
2. There shall not be any adverse consequences when running against hdfs or local fs.
### Query engine
None
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: Improve HadoopFileIO performance when working with cloud storage [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15353:
URL: https://github.com/apache/iceberg/issues/15353#issuecomment-5577190457
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Optimise HadoopFileIO for cloud IO [iceberg]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #15353:
URL: https://github.com/apache/iceberg/issues/15353#issuecomment-3926581283
Not going to be able measure this in tests unless some go into iceberg-aws, which would make for a much bigger PR.
The other tunings would be
* class AdaptingIterator to implement CloseableIterator and pass down the close() call if the inner iterator asks for it. This is because s3a lists prefetch; a close() cancels the operation.
* look at createFile for file creation and what operations are good there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: Improve HadoopFileIO performance when working with cloud storage [iceberg]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #15353:
URL: https://github.com/apache/iceberg/issues/15353#issuecomment-5601956979
leaving to others; I'm not working on big data stack any more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org