You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jfuechsl (via GitHub)" <gi...@apache.org> on 2023/05/28 07:47:42 UTC
[GitHub] [arrow-rs] jfuechsl opened a new issue, #4301: [Question - object_store] AWS credentials provider
jfuechsl opened a new issue, #4301:
URL: https://github.com/apache/arrow-rs/issues/4301
**Which part is this question about**
`object_store` crate.
**Describe your question**
The `AmazonS3Builder` implements its own AWS credentials resolution and provisioning (which currently doesn't support ECS environments natively).
AWS SDKs (see [here](https://github.com/awslabs/aws-sdk-rust/blob/d1962fd42d376c5c695877041d2a269fd1fa50cd/sdk/aws-config/src/default_provider/credentials.rs#L25) for the Rust SDK) feature a well defined Credential Provider Chain.
My question is: why does this project not use the official AWS Rust SDK implementation for credentials (and possibly the S3 client implementation)?
**Additional context**
Issue #4283 should fix credentials resolution for ECS runtime environments. However the fix doesn't work, since the provided metadata_endpoint is still used in the context of the EC2 credentials provider which postfixes a `/latest/api/token` path, which breaks credentials resolution for ECS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold closed issue #4301: [Question - object_store] AWS credentials provider
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #4301: [Question - object_store] AWS credentials provider
URL: https://github.com/apache/arrow-rs/issues/4301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #4301: [Question - object_store] AWS credentials provider
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4301:
URL: https://github.com/apache/arrow-rs/issues/4301#issuecomment-1573884513
`label_issue.py` automatically added labels {'object-store'} from #4310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #4301: [Question - object_store] AWS credentials provider
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4301:
URL: https://github.com/apache/arrow-rs/issues/4301#issuecomment-1566264595
There is more historical context on https://github.com/apache/arrow-rs/issues/2176 but the high level reason is because using the cloud provider SDKs resulted in significant dependency bloat, inconsistent behaviour w.r.t errors, retries, etc... and missing features.
Its also worth noting the AWS SDK you reference didn't exist when this crate was originally written, we used rusoto instead, and is still marked not suitable for production use.
I was under the impression the ECS credentials were now fixed, albeit not released, but a dependency patch should allow using the fixed version based on git sha? Is there something still not working correctly?
There is also the escape valve of https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_credentials if you wish to source credentials through some other mechanism
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jfuechsl commented on issue #4301: [Question - object_store] AWS credentials provider
Posted by "jfuechsl (via GitHub)" <gi...@apache.org>.
jfuechsl commented on issue #4301:
URL: https://github.com/apache/arrow-rs/issues/4301#issuecomment-1568243904
Understood, thanks for the elaboration.
Unfortunately, I realised that the ECS credentials fix does not work. The reason is that the metadata endpoint is still being used in the context of the EC2 credentials provider `InstanceCredentialProvider`, which further appends a path to the endpoint https://github.com/apache/arrow-rs/blob/ea008922445d84d957cf3f89df793187c22d82d8/object_store/src/aws/credential.rs#L429
A proper fix would need to adapt the credentials resolution logic in `AmazonS3Builder::build` by including a ECS specific provider (thus mirroring the official resolution chain in the AWS SDKs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #4301: [Question - object_store] AWS credentials provider
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4301:
URL: https://github.com/apache/arrow-rs/issues/4301#issuecomment-1568281293
Thank you for reporting, I'll get a fix in for you today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org