You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "litiliu (via GitHub)" <gi...@apache.org> on 2026/06/29 07:06:06 UTC
[PR] Support AWS S3 credentials provider mode [fluss]
litiliu opened a new pull request, #3540:
URL: https://github.com/apache/fluss/pull/3540
<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
- Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
- **Generative AI disclosure:** Indicate whether generative AI tools were used in authoring this PR. If yes, specify the tool below.
- [ ] No generative AI tools used
- [ ] Yes (please specify the tool below)
**(The sections below can be removed for hotfixes or typos)**
-->
<!--
Generated-by: [Tool Name and Version] following [the guidelines](https://github.com/apache/fluss/blob/main/AGENTS.md)
-->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Brief change log
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] Support AWS S3 credentials provider mode [fluss]
Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on code in PR #3540:
URL: https://github.com/apache/fluss/pull/3540#discussion_r3526798711
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -147,6 +168,33 @@ private AWSSecurityTokenService buildStsClient() {
return builder.build();
}
+ @Nullable
+ AWSCredentialsProvider createStsCredentialsProvider() {
+ if (credentialProviderList != null) {
+ AWSCredentials credentials = credentialProviderList.getCredentials();
+ checkArgument(
Review Comment:
Added a note to the IRSA/instance profile section clarifying that the explicit server-side provider mode only supports long-term credentials and cannot be combined with s3.assumed.role.arn. IRSA and instance profile providers should continue to use the default AWS credential chain. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] Support AWS S3 credentials provider mode [fluss]
Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on code in PR #3540:
URL: https://github.com/apache/fluss/pull/3540#discussion_r3526798711
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -147,6 +168,33 @@ private AWSSecurityTokenService buildStsClient() {
return builder.build();
}
+ @Nullable
+ AWSCredentialsProvider createStsCredentialsProvider() {
+ if (credentialProviderList != null) {
+ AWSCredentials credentials = credentialProviderList.getCredentials();
+ checkArgument(
Review Comment:
Added a note to the IRSA/instance profile section clarifying that the explicit server-side provider mode only supports long-term credentials and cannot be combined with s3.assumed.role.arn. IRSA and instance profile providers should continue to use the default AWS credential chain. Thanks!Added a note to the IRSA/instance profile section clarifying that the explicit server-side provider mode only supports long-term credentials and cannot be combined with s3.assumed.role.arn. IRSA and instance profile providers should continue to use the default AWS credential chain. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] Support AWS S3 credentials provider mode [fluss]
Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on code in PR #3540:
URL: https://github.com/apache/fluss/pull/3540#discussion_r3526802796
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/S3FileSystemPlugin.java:
##########
@@ -23,14 +23,17 @@
import org.apache.fluss.fs.FileSystem;
import org.apache.fluss.fs.FileSystemPlugin;
import org.apache.fluss.fs.s3.token.S3ADelegationTokenReceiver;
+import org.apache.fluss.fs.s3.token.S3DelegationTokenProvider;
import org.apache.fluss.fs.s3.token.S3DelegationTokenReceiver;
+import org.apache.commons.lang3.StringUtils;
Review Comment:
Updated this one as well to use Fluss's StringUtils.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] Support AWS S3 credentials provider mode [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #3540:
URL: https://github.com/apache/fluss/pull/3540#discussion_r3520836625
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -63,21 +73,31 @@ public class S3DelegationTokenProvider {
@Nullable private final String secretKey;
@Nullable private final String roleArn;
@Nullable private final String stsEndpoint;
+ @Nullable private final AWSCredentialProviderList credentialProviderList;
private final Map<String, String> additionInfos;
- public S3DelegationTokenProvider(String scheme, Configuration conf) {
+ public S3DelegationTokenProvider(String scheme, Configuration conf) throws IOException {
this.scheme = scheme;
this.region = conf.get(REGION_KEY);
checkArgument(region != null, "Region is not set.");
this.accessKey = conf.get(ACCESS_KEY_ID);
this.secretKey = conf.get(ACCESS_KEY_SECRET);
this.roleArn = conf.get(ROLE_ARN_KEY);
this.stsEndpoint = conf.get(STS_ENDPOINT_KEY);
+ boolean hasCredentialProvider =
+ conf.getBoolean(CREDENTIAL_PROVIDER_EXPLICITLY_CONFIGURED, false)
+ && StringUtils.isNotBlank(conf.getTrimmed(AWS_CREDENTIALS_PROVIDER));
checkArgument(
(accessKey == null) == (secretKey == null),
"S3 access key and secret key must both be set or both be unset.");
- if (accessKey == null) {
+ if (hasCredentialProvider && roleArn != null) {
+ throw new IllegalArgumentException(
+ "AssumeRole and a custom AWS credentials provider cannot be configured together.");
+ }
+ this.credentialProviderList =
Review Comment:
The description says configuring DynamicTemporaryAWSCredentialsProvider for server mode is "rejected", but there's no guard, so it just gets instantiated here and throws NoAwsCredentialsException at the first token request. Can you clarify?
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/S3FileSystemPlugin.java:
##########
@@ -23,14 +23,17 @@
import org.apache.fluss.fs.FileSystem;
import org.apache.fluss.fs.FileSystemPlugin;
import org.apache.fluss.fs.s3.token.S3ADelegationTokenReceiver;
+import org.apache.fluss.fs.s3.token.S3DelegationTokenProvider;
import org.apache.fluss.fs.s3.token.S3DelegationTokenReceiver;
+import org.apache.commons.lang3.StringUtils;
Review Comment:
ditto
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -147,6 +168,33 @@ private AWSSecurityTokenService buildStsClient() {
return builder.build();
}
+ @Nullable
+ AWSCredentialsProvider createStsCredentialsProvider() {
+ if (credentialProviderList != null) {
+ AWSCredentials credentials = credentialProviderList.getCredentials();
+ checkArgument(
Review Comment:
This is effectively the "long-term creds only" gate, and it's the thing people will trip on instance profiles/IRSA roles all return session creds and land here at token time (lazily).
Given s3.md already has an IRSA section, can we add a short note there for this new mode: long-term creds only, not compatible with AssumeRole? Otherwise it's a bit hidden for operational usage
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -29,12 +32,16 @@
import com.amazonaws.services.securitytoken.model.AssumeRoleResult;
import com.amazonaws.services.securitytoken.model.Credentials;
import com.amazonaws.services.securitytoken.model.GetSessionTokenResult;
+import org.apache.commons.lang3.StringUtils;
Review Comment:
commons-lang3 is only transitive here, shall we use org.apache.commons.lang3.StringUtils?
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -63,21 +73,31 @@ public class S3DelegationTokenProvider {
@Nullable private final String secretKey;
@Nullable private final String roleArn;
@Nullable private final String stsEndpoint;
+ @Nullable private final AWSCredentialProviderList credentialProviderList;
private final Map<String, String> additionInfos;
- public S3DelegationTokenProvider(String scheme, Configuration conf) {
+ public S3DelegationTokenProvider(String scheme, Configuration conf) throws IOException {
this.scheme = scheme;
this.region = conf.get(REGION_KEY);
checkArgument(region != null, "Region is not set.");
this.accessKey = conf.get(ACCESS_KEY_ID);
this.secretKey = conf.get(ACCESS_KEY_SECRET);
this.roleArn = conf.get(ROLE_ARN_KEY);
this.stsEndpoint = conf.get(STS_ENDPOINT_KEY);
+ boolean hasCredentialProvider =
+ conf.getBoolean(CREDENTIAL_PROVIDER_EXPLICITLY_CONFIGURED, false)
+ && StringUtils.isNotBlank(conf.getTrimmed(AWS_CREDENTIALS_PROVIDER));
checkArgument(
(accessKey == null) == (secretKey == null),
"S3 access key and secret key must both be set or both be unset.");
- if (accessKey == null) {
+ if (hasCredentialProvider && roleArn != null) {
Review Comment:
Since we're adding all these checks in this PR - this one fails fast, but the session/empty-cred cases only fail on the first token request.
Could we check those at construction too, so they're consistent? Not blocking though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] Support AWS S3 credentials provider mode [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni merged PR #3540:
URL: https://github.com/apache/fluss/pull/3540
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] Support AWS S3 credentials provider mode [fluss]
Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on code in PR #3540:
URL: https://github.com/apache/fluss/pull/3540#discussion_r3526751256
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -63,21 +73,31 @@ public class S3DelegationTokenProvider {
@Nullable private final String secretKey;
@Nullable private final String roleArn;
@Nullable private final String stsEndpoint;
+ @Nullable private final AWSCredentialProviderList credentialProviderList;
private final Map<String, String> additionInfos;
- public S3DelegationTokenProvider(String scheme, Configuration conf) {
+ public S3DelegationTokenProvider(String scheme, Configuration conf) throws IOException {
this.scheme = scheme;
this.region = conf.get(REGION_KEY);
checkArgument(region != null, "Region is not set.");
this.accessKey = conf.get(ACCESS_KEY_ID);
this.secretKey = conf.get(ACCESS_KEY_SECRET);
this.roleArn = conf.get(ROLE_ARN_KEY);
this.stsEndpoint = conf.get(STS_ENDPOINT_KEY);
+ boolean hasCredentialProvider =
+ conf.getBoolean(CREDENTIAL_PROVIDER_EXPLICITLY_CONFIGURED, false)
+ && StringUtils.isNotBlank(conf.getTrimmed(AWS_CREDENTIALS_PROVIDER));
checkArgument(
(accessKey == null) == (secretKey == null),
"S3 access key and secret key must both be set or both be unset.");
- if (accessKey == null) {
+ if (hasCredentialProvider && roleArn != null) {
Review Comment:
Thanks for the suggestion. S3DelegationTokenProvider itself is lazily constructed during the first obtainSecurityToken() call, so resolving credentials in the constructor would not move the failure to server startup; it would only perform the validation earlier within the same request. Since the credentials are already validated before calling STS and this is non-blocking, I'd prefer to keep the resolution lazy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] Support AWS S3 credentials provider mode [fluss]
Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on code in PR #3540:
URL: https://github.com/apache/fluss/pull/3540#discussion_r3526800801
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -29,12 +32,16 @@
import com.amazonaws.services.securitytoken.model.AssumeRoleResult;
import com.amazonaws.services.securitytoken.model.Credentials;
import com.amazonaws.services.securitytoken.model.GetSessionTokenResult;
+import org.apache.commons.lang3.StringUtils;
Review Comment:
Good point. I replaced org.apache.commons.lang3.StringUtils with Fluss's org.apache.fluss.utils.StringUtils, so we no longer rely on the transitive commons-lang3 dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] Support AWS S3 credentials provider mode [fluss]
Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on code in PR #3540:
URL: https://github.com/apache/fluss/pull/3540#discussion_r3526666987
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -63,21 +73,31 @@ public class S3DelegationTokenProvider {
@Nullable private final String secretKey;
@Nullable private final String roleArn;
@Nullable private final String stsEndpoint;
+ @Nullable private final AWSCredentialProviderList credentialProviderList;
private final Map<String, String> additionInfos;
- public S3DelegationTokenProvider(String scheme, Configuration conf) {
+ public S3DelegationTokenProvider(String scheme, Configuration conf) throws IOException {
this.scheme = scheme;
this.region = conf.get(REGION_KEY);
checkArgument(region != null, "Region is not set.");
this.accessKey = conf.get(ACCESS_KEY_ID);
this.secretKey = conf.get(ACCESS_KEY_SECRET);
this.roleArn = conf.get(ROLE_ARN_KEY);
this.stsEndpoint = conf.get(STS_ENDPOINT_KEY);
+ boolean hasCredentialProvider =
+ conf.getBoolean(CREDENTIAL_PROVIDER_EXPLICITLY_CONFIGURED, false)
+ && StringUtils.isNotBlank(conf.getTrimmed(AWS_CREDENTIALS_PROVIDER));
checkArgument(
(accessKey == null) == (secretKey == null),
"S3 access key and secret key must both be set or both be unset.");
- if (accessKey == null) {
+ if (hasCredentialProvider && roleArn != null) {
+ throw new IllegalArgumentException(
+ "AssumeRole and a custom AWS credentials provider cannot be configured together.");
+ }
+ this.credentialProviderList =
Review Comment:
Good catch. DynamicTemporaryAWSCredentialsProvider is an internal provider used for client-side delegated credentials and is not intended to be configured manually as a server-side credential source. However, since the configuration accepts arbitrary provider class names, I've added an explicit check to reject it before constructing the S3A provider list, along with a regression test. It now fails with a clear configuration error instead of failing on the first credential request. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] Support AWS S3 credentials provider mode [fluss]
Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on PR #3540:
URL: https://github.com/apache/fluss/pull/3540#issuecomment-4889743135
@fresh-borzoni Thanks for the review! I've addressed the comments, added the S3 credential-provider documentation, and rebased the branch onto the latest main. PTAL when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org