You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "phillipleblanc (via GitHub)" <gi...@apache.org> on 2025/07/19 13:26:54 UTC
[PR] Add support for custom credential loader for S3 FileIO [iceberg-rust]
phillipleblanc opened a new pull request, #1528:
URL: https://github.com/apache/iceberg-rust/pull/1528
## Which issue does this PR close?
- Closes #1527
## What changes are included in this PR?
Adds the ability to provide custom extensions to the `FileIOBuilder`. Currently the only supported extension is `CustomAwsCredentialLoader` which is a newtype around [`AwsCredentialLoad`](https://docs.rs/reqsign/0.16.3/reqsign/trait.AwsCredentialLoad.html), which is what OpenDAL expects.
I've added extensions to the `RestCatalog` as well, and when its constructing `FileIO` for table operations, passes along any defined extensions into the `FileIOBuilder`, which then get passed into the underlying OpenDAL constructors.
## Are these changes tested?
Yes, tests added in `crates/iceberg/tests/file_io_s3_test.rs` that verify the extension is working.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for custom credential loader for S3 FileIO [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #1528:
URL: https://github.com/apache/iceberg-rust/pull/1528#issuecomment-3599659055
I think we will need to remove this once we support Storage trait, so users could use a custom Storage implementation to provide credentials.
The main problem right now is `Extensions` is not serializable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for custom credential loader for S3 FileIO [iceberg-rust]
Posted by "phillipleblanc (via GitHub)" <gi...@apache.org>.
phillipleblanc commented on PR #1528:
URL: https://github.com/apache/iceberg-rust/pull/1528#issuecomment-3609425606
> I think we will need to remove this once we support Storage trait, so users could use a custom Storage implementation to provide credentials.
>
> The main problem right now is `Extensions` is not serializable
That makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for custom credential loader for S3 FileIO [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #1528:
URL: https://github.com/apache/iceberg-rust/pull/1528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for custom credential loader for S3 FileIO [iceberg-rust]
Posted by "phillipleblanc (via GitHub)" <gi...@apache.org>.
phillipleblanc commented on PR #1528:
URL: https://github.com/apache/iceberg-rust/pull/1528#issuecomment-3096442793
> This is another motivating case of #1314
Ah yeah, I wasn't aware of that discussion - thanks for the link! I represent another case where I'm already heavily invested in the `object_store` ecosystem and would like to be able to use that to power my Iceberg reads/writes in DataFusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for custom credential loader for S3 FileIO [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #1528:
URL: https://github.com/apache/iceberg-rust/pull/1528#issuecomment-3096075820
This is another motivating case of https://github.com/apache/iceberg-rust/issues/1314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org