You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "jackye1995 (via GitHub)" <gi...@apache.org> on 2025/08/22 22:01:34 UTC
[PR] feat(services/s3): add container credentials support for ECS and EKS [opendal]
jackye1995 opened a new pull request, #6525:
URL: https://github.com/apache/opendal/pull/6525
## Summary
This PR adds support for AWS container credentials used in ECS tasks and EKS pods, enabling OpenDAL to seamlessly work in containerized AWS environments.
## Background
Container credentials are a key feature for AWS workloads running in ECS and EKS:
- **ECS Task IAM Roles**: Allow containers to assume IAM roles without embedding credentials
- **EKS Pod Identity**: Enable pods to authenticate using service account tokens
This feature addresses issue #6456 by implementing the container credentials portion of the AWS configuration compatibility improvements.
## Changes
### New Configuration Fields
| Field | Aliases | Purpose |
|-------|---------|---------|
| `container_credentials_relative_uri` | `aws_container_credentials_relative_uri` | ECS task IAM role endpoint |
| `container_credentials_full_uri` | `aws_container_credentials_full_uri` | EKS pod identity endpoint |
| `container_authorization_token_file` | `aws_container_authorization_token_file` | EKS service account token file |
### Builder Methods
Added corresponding builder methods for programmatic configuration:
- `container_credentials_relative_uri(uri: &str)`
- `container_credentials_full_uri(uri: &str)`
- `container_authorization_token_file(file: &str)`
### Example Usage
```rust
// ECS Task IAM Role
let s3 = S3::default()
.bucket("my-bucket")
.container_credentials_relative_uri("/v2/credentials/12345678-1234-1234-1234-123456789012")
.build()?;
// EKS Pod Identity
let s3 = S3::default()
.bucket("my-bucket")
.container_credentials_full_uri("https://localhost:1234/token")
.container_authorization_token_file("/var/run/secrets/eks.amazonaws.com/serviceaccount/token")
.build()?;
```
### JSON Configuration
```json
{
"bucket": "my-bucket",
"aws_container_credentials_relative_uri": "/v2/credentials/12345678-1234-1234-1234-123456789012"
}
```
## Implementation Details
### Current State
- ✅ Configuration fields with serde aliases
- ✅ Builder methods for programmatic setup
- ✅ Comprehensive test coverage
- ✅ Debug logging when container credentials are configured
- 🔄 Placeholder implementation (credential loading will be added in follow-up PR)
### Design Decisions
1. **Serde Aliases**: Following the same pattern as the previous PR (#6524) to support both OpenDAL and AWS SDK naming conventions
2. **Credential Provider Pattern**: Implementation follows Apache Arrow's object_store approach:
- ECS: HTTP GET to `http://169.254.170.2{relative_uri}`
- EKS: HTTP GET to `{full_uri}` with `Authorization: {token}` header
- Both return JSON with `AccessKeyId`, `SecretAccessKey`, `Token`, `Expiration`
3. **Placeholder Implementation**: The current implementation detects and logs container credential configuration but uses a TODO placeholder for the actual credential loading. This allows:
- Configuration validation and testing
- Integration with existing customized credential loaders
- Future implementation of automatic credential loading
## Testing
- ✅ Unit tests for configuration deserialization with aliases
- ✅ All existing S3 tests pass (16/16)
- ✅ Clippy and formatting checks pass
- ✅ No breaking changes to existing functionality
## Future Work
The next PR will implement the actual credential loading logic by:
1. Creating proper `AwsCredentialLoad` implementations
2. Adding HTTP-based credential fetching
3. Implementing credential caching and refresh logic
4. Adding integration tests with mock endpoints
## Compatibility
This change is fully backward compatible. Existing configurations continue to work unchanged, with new fields providing additional flexibility for containerized environments.
Related to #6456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/s3): add container credentials support for ECS and EKS [opendal]
Posted by "rohankmr414 (via GitHub)" <gi...@apache.org>.
rohankmr414 commented on PR #6525:
URL: https://github.com/apache/opendal/pull/6525#issuecomment-3755584163
@Xuanwo - now that the reqsign ECS support has been merged and released in v0.17.0, could this PR be reviewed? We're blocked on this for EKS Pod Identity support in [iceberg-rust](https://github.com/apache/iceberg-rust) with REST catalog. Happy to help test!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/s3): add container credentials support for ECS and EKS [opendal]
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #6525:
URL: https://github.com/apache/opendal/pull/6525#issuecomment-3216211971
Pending merge of https://github.com/Xuanwo/reqsign/pull/598 and upgrading reqsign
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/s3): add container credentials support for ECS and EKS [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #6525:
URL: https://github.com/apache/opendal/pull/6525#issuecomment-3766834449
> @Xuanwo - now that the reqsign ECS support has been merged and released in v0.17.0, could this PR be reviewed? We're blocked on this for EKS Pod Identity support in [iceberg-rust](https://github.com/apache/iceberg-rust) with REST catalog. Happy to help test!
Thank you for the reminder! @jackye1995, are you interested in updating this PR and rebasing it on the new reqsign API for ECS support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/s3): add container credentials support for ECS and EKS [opendal]
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #6525:
URL: https://github.com/apache/opendal/pull/6525#issuecomment-3215823601
Looks like the credentials provider should go to reqsign, will create a PR there first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/s3): add container credentials support for ECS and EKS [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #6525:
URL: https://github.com/apache/opendal/pull/6525#issuecomment-4509985568
Oh, I understand this PR in wrong. cc @jackye1995, do you want to resolve the conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/s3): add container credentials support for ECS and EKS [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed pull request #6525: feat(services/s3): add container credentials support for ECS and EKS
URL: https://github.com/apache/opendal/pull/6525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/s3): add container credentials support for ECS and EKS [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #6525:
URL: https://github.com/apache/opendal/pull/6525#issuecomment-4509976840
Should already supported by the new reqsign.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org