You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Samrose-Ahmed (via GitHub)" <gi...@apache.org> on 2024/07/16 08:51:08 UTC
[I] new feature: support AWS Container credential provider [opendal]
Samrose-Ahmed opened a new issue, #4903:
URL: https://github.com/apache/opendal/issues/4903
### Feature Description
via `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` environment variable in ECS/container settings: see https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html
### Problem and Solution
load credentials in ecs environment
### Additional Context
_No response_
### Are you willing to contribute to the development of this feature?
- [ ] Yes, I am willing to contribute to the development of this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: support AWS Container credential provider [opendal]
Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.
michaelrommel commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-3839989099
Here is a temporary solution:
```rust
use aws_config::meta::region::RegionProviderChain;
use aws_credential_types::provider::ProvideCredentials;
use aws_types::SdkConfig;
struct AwsCredentialLoad;
#[async_trait::async_trait]
impl reqsign::AwsCredentialLoad for AwsCredentialLoad {
async fn load_credential(&self, _client: Client) -> anyhow::Result<Option<AwsCredential>> {
println!("Loading AWS credentials");
let region_provider = RegionProviderChain::default_provider().or_else("us-east-1");
let config: SdkConfig = aws_config::defaults(BehaviorVersion::v2026_01_12())
.region(region_provider)
.load()
.await;
if let Some(credentials_provider) = config.credentials_provider() {
let credentials = credentials_provider.provide_credentials().await?;
let duration = credentials
.expiry()
.unwrap()
.duration_since(UNIX_EPOCH)
.expect("SystemTime is before UNIX_EPOCH");
let expiry = Utc
.timestamp_opt(duration.as_secs() as i64, duration.subsec_nanos())
.single();
println!("AWS Token expires at: {:?}", expiry);
return Ok(Some(AwsCredential {
access_key_id: credentials.access_key_id().to_string(),
secret_access_key: credentials.secret_access_key().to_string(),
session_token: credentials.session_token().map(|s| s.to_string()),
expires_in: expiry,
}));
}
Ok(None)
}
}
let builder = S3::default()
.customized_credential_load(Box::new(AwsCredentialLoad))
.endpoint("https://s3.amazonaws.com")
.region(®ion) // String with the region name
.bucket(&bucket) // String with the bucket name
.root("/");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: support AWS Container credential provider [opendal]
Posted by "Samrose-Ahmed (via GitHub)" <gi...@apache.org>.
Samrose-Ahmed commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2270660705
Yes it's the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: support AWS Container credential provider [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2270672551
Hi, @Samrose-Ahmed, I expect to add this feature in our next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: support AWS Container credential provider [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2270670211
Thank you! I will redirect the discussion here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: support AWS Container credential provider [opendal]
Posted by "matthias-Q (via GitHub)" <gi...@apache.org>.
matthias-Q commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2988524683
I stumbled over this issue today. Here is a workaround with a start script that sets the ENV vars when running inside ECS
```bash
#!/bin/bash
CREDS_URI=${AWS_CONTAINER_CREDENTIALS_FULL_URI:-"http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}"}
CREDS_RESPONSE=$(curl -s $CREDS_URI)
export AWS_ACCESS_KEY_ID=$(echo $CREDS_RESPONSE | jq -r '.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $CREDS_RESPONSE | jq -r '.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $CREDS_RESPONSE | jq -r '.Token')
exec "$@"
```
I hope OpenDAL will support the proper Credentials Chain in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: support AWS Container credential provider [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2282750538
> General question regarding this, would it be a good idea to wrap the AWS SDK with the credentials provider to do this job? This has been my current workaround for this issue and might be a better long-term solution for managing all the various ways you can authenticate with AWS.
Thanks for the suggestion. We previously had a discussion here: https://github.com/apache/opendal/discussions/139.
The various methods for authentication are limited and rarely change, so we can ultimately support them all. I believe it isn't worth introducing the AWS SDK for authentication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: support AWS Container credential provider [opendal]
Posted by "ChillFish8 (via GitHub)" <gi...@apache.org>.
ChillFish8 commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2282739252
General question regarding this, would it be a good idea to wrap the AWS SDK with the credentials provider to do this job? This has been my current workaround for this issue and might be a better long-term solution for managing all the various ways you can authenticate with AWS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: support AWS Container credential provider [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2270626701
Hi, @Samrose-Ahmed, does this issue related to https://github.com/apache/opendal/issues/4964?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: support AWS Container credential provider [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2989950870
Hi, thank you @matthias-Q for this workaround. I'm currently **very active** in the project https://github.com/Xuanwo/reqsign, which is the crate that opendal relies on for this feature. I plan to integrate these changes in the next major version of opendal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: support AWS Container credential provider [opendal]
Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.
michaelrommel commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-3822821509
Hello @Xuanwo ,
Is it possible to update the status here? I try to access S3 buckets from a Fargate Container and cannot get the credentials automatically...
Thanks,
Michael.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org