You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Samrose-Ahmed (via GitHub)" <gi...@apache.org> on 2024/07/16 08:51:08 UTC

[I] new feature: support AWS Container credential provider [opendal]

Samrose-Ahmed opened a new issue, #4903:
URL: https://github.com/apache/opendal/issues/4903

   ### Feature Description
   
   via `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` environment variable in ECS/container settings: see https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html
   
   ### Problem and Solution
   
   load credentials in ecs environment
   
   ### Additional Context
   
   _No response_
   
   ### Are you willing to contribute to the development of this feature?
   
   - [ ] Yes, I am willing to contribute to the development of this feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: support AWS Container credential provider [opendal]

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.
michaelrommel commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-3839989099

   Here is a temporary solution:
   
   ```rust
   use aws_config::meta::region::RegionProviderChain;
   use aws_credential_types::provider::ProvideCredentials;
   use aws_types::SdkConfig;
   
   struct AwsCredentialLoad;
   
   #[async_trait::async_trait]
   impl reqsign::AwsCredentialLoad for AwsCredentialLoad {
       async fn load_credential(&self, _client: Client) -> anyhow::Result<Option<AwsCredential>> {
           println!("Loading AWS credentials");
           let region_provider = RegionProviderChain::default_provider().or_else("us-east-1");
   
           let config: SdkConfig = aws_config::defaults(BehaviorVersion::v2026_01_12())
               .region(region_provider)
               .load()
               .await;
   
           if let Some(credentials_provider) = config.credentials_provider() {
               let credentials = credentials_provider.provide_credentials().await?;
               let duration = credentials
                   .expiry()
                   .unwrap()
                   .duration_since(UNIX_EPOCH)
                   .expect("SystemTime is before UNIX_EPOCH");
               let expiry = Utc
                   .timestamp_opt(duration.as_secs() as i64, duration.subsec_nanos())
                   .single();
               println!("AWS Token expires at: {:?}", expiry);
               return Ok(Some(AwsCredential {
                   access_key_id: credentials.access_key_id().to_string(),
                   secret_access_key: credentials.secret_access_key().to_string(),
                   session_token: credentials.session_token().map(|s| s.to_string()),
                   expires_in: expiry,
               }));
           }
           Ok(None)
       }
   }
   
      let builder = S3::default()
           .customized_credential_load(Box::new(AwsCredentialLoad))
           .endpoint("https://s3.amazonaws.com")
           .region(&region) // String with the region name
           .bucket(&bucket) // String with the bucket name
           .root("/");
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: support AWS Container credential provider [opendal]

Posted by "Samrose-Ahmed (via GitHub)" <gi...@apache.org>.
Samrose-Ahmed commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2270660705

   Yes it's the same.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: support AWS Container credential provider [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2270672551

   Hi, @Samrose-Ahmed, I expect to add this feature in our next release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: support AWS Container credential provider [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2270670211

   Thank you! I will redirect the discussion here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: support AWS Container credential provider [opendal]

Posted by "matthias-Q (via GitHub)" <gi...@apache.org>.
matthias-Q commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2988524683

   I stumbled over this issue today. Here is a workaround with a start script that sets the ENV vars when running inside ECS
   ```bash
   #!/bin/bash
   
   CREDS_URI=${AWS_CONTAINER_CREDENTIALS_FULL_URI:-"http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}"}
   CREDS_RESPONSE=$(curl -s $CREDS_URI)
   
   export AWS_ACCESS_KEY_ID=$(echo $CREDS_RESPONSE | jq -r '.AccessKeyId')
   export AWS_SECRET_ACCESS_KEY=$(echo $CREDS_RESPONSE | jq -r '.SecretAccessKey')
   export AWS_SESSION_TOKEN=$(echo $CREDS_RESPONSE | jq -r '.Token')
   
   exec "$@"
   ```
   
   I hope OpenDAL will support the proper Credentials Chain in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: support AWS Container credential provider [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2282750538

   > General question regarding this, would it be a good idea to wrap the AWS SDK with the credentials provider to do this job? This has been my current workaround for this issue and might be a better long-term solution for managing all the various ways you can authenticate with AWS.
   
   Thanks for the suggestion. We previously had a discussion here: https://github.com/apache/opendal/discussions/139.
   
   The various methods for authentication are limited and rarely change, so we can ultimately support them all. I believe it isn't worth introducing the AWS SDK for authentication.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: support AWS Container credential provider [opendal]

Posted by "ChillFish8 (via GitHub)" <gi...@apache.org>.
ChillFish8 commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2282739252

   General question regarding this, would it be a good idea to wrap the AWS SDK with the credentials provider to do this job? This has been my current workaround for this issue and might be a better long-term solution for managing all the various ways you can authenticate with AWS.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: support AWS Container credential provider [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2270626701

   Hi, @Samrose-Ahmed, does this issue related to https://github.com/apache/opendal/issues/4964?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: support AWS Container credential provider [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-2989950870

   Hi, thank you @matthias-Q for this workaround. I'm currently **very active** in the project https://github.com/Xuanwo/reqsign, which is the crate that opendal relies on for this feature. I plan to integrate these changes in the next major version of opendal.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: support AWS Container credential provider [opendal]

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.
michaelrommel commented on issue #4903:
URL: https://github.com/apache/opendal/issues/4903#issuecomment-3822821509

   Hello @Xuanwo ,
   
   Is it possible to update the status here? I try to access S3 buckets from a Fargate Container and cannot get the credentials automatically...
   
   Thanks,
   
     Michael.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org