You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "prodriguezdefino (via GitHub)" <gi...@apache.org> on 2025/03/26 19:54:36 UTC
[PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
prodriguezdefino opened a new pull request, #34440:
URL: https://github.com/apache/beam/pull/34440
WIP
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2783132734
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @ahmedabu98 for label java.
R: @chamikaramj for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2756210584
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @m-trieu for label java.
R: @chamikaramj for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34440:
URL: https://github.com/apache/beam/pull/34440#discussion_r2080822379
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/auth/providers/StsAssumeRoleWithDynamicWebIdentityCredentialsProvider.java:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.common.auth.providers;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkState;
+
+import com.fasterxml.jackson.annotation.JsonSubTypes;
+import com.google.auto.value.AutoValue;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.function.Supplier;
+import javax.annotation.Nullable;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Suppliers;
+import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.auth.StsAssumeRoleWithWebIdentityCredentialsProvider;
+import software.amazon.awssdk.services.sts.model.AssumeRoleWithWebIdentityRequest;
+import software.amazon.awssdk.utils.SdkAutoCloseable;
+
+/**
+ * An implementation of AwsCredentialsProvider that periodically sends an {@link
+ * AssumeRoleWithWebIdentityRequest} to the AWS Security Token Service to maintain short-lived
+ * sessions to use for authentication. In particular this class will use a {@link
+ * StsAssumeRoleWithWebIdentityCredentialsProvider} instance as a delegate for the actual
+ * implementation but it takes care of retrieving a refreshed web id token with every credential's
+ * resolution request. This is created using builder().
+ *
+ * @see <a
+ * href="https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html">API
+ * reference</a>
+ */
+@AutoValue
+@JsonSubTypes({
+ @JsonSubTypes.Type(
+ value = AutoValue_StsAssumeRoleWithDynamicWebIdentityCredentialsProvider.class,
+ name = "StsAssumeRoleWithDynamicWebIdentityCredentialsProvider")
+})
+public abstract class StsAssumeRoleWithDynamicWebIdentityCredentialsProvider
+ implements AwsCredentialsProvider, SdkAutoCloseable {
+
+ public static final Integer DEFAULT_SESSION_DURATION_SECS = 3600;
+
+ abstract CredentialsProviderDelegate credentialsProviderDelegate();
+
+ public abstract String audience();
+
+ public abstract String assumedRoleArn();
+
+ public abstract String webIdTokenProviderFQCN();
+
+ @Nullable
+ public abstract Integer sessionDurationSecs();
+
+ @Override
+ public AwsCredentials resolveCredentials() {
+ return credentialsProviderDelegate().resolveCredentials();
+ }
+
+ @Override
+ public void close() {
+ credentialsProviderDelegate().close();
+ }
+
+ /**
+ * Creates a builder for the type {@link StsAssumeRoleWithDynamicWebIdentityCredentialsProvider}.
+ *
+ * @return an initialized builder instance.
+ */
+ public static StsAssumeRoleWithDynamicWebIdentityCredentialsProvider.Builder builder() {
+ return new AutoValue_StsAssumeRoleWithDynamicWebIdentityCredentialsProvider.Builder();
+ }
+
+ /** Builder class for {@link StsAssumeRoleWithDynamicWebIdentityCredentialsProvider}. */
+ @AutoValue.Builder
+ public abstract static class Builder {
+
+ /**
+ * Sets the role to be assumed by the authentication request.
+ *
+ * @param roleArn the AWS role ARN.
+ * @return this builder instance.
+ */
+ public abstract Builder setAssumedRoleArn(String roleArn);
+
+ /**
+ * Sets the audience to be used for the web id token request.
+ *
+ * @param audience the audience value.
+ * @return this builder instance.
+ */
+ public abstract Builder setAudience(String audience);
+
+ /**
+ * The fully qualified class name for the web id token provider. The class should be accessible
+ * in the classpath.
+ *
+ * @param idTokenProviderFQCN the class name.
+ * @return this builder instance.
+ */
+ public abstract Builder setWebIdTokenProviderFQCN(String idTokenProviderFQCN);
+
+ /**
+ * The session duration in seconds for the authentication request, by default this value is
+ * 3600.
+ *
+ * @param durationSecs the duration in seconds.
+ * @return this builder instance.
+ */
+ public abstract Builder setSessionDurationSecs(@Nullable Integer durationSecs);
+
+ /**
+ * For testing purposes, should not be set in a production setup.
+ *
+ * @param delegate The testing delegate to use.
+ * @return this builder instance.
+ */
+ abstract Builder setCredentialsProviderDelegate(CredentialsProviderDelegate delegate);
+
+ abstract Optional<CredentialsProviderDelegate> credentialsProviderDelegate();
+
+ abstract String assumedRoleArn();
+
+ abstract String audience();
+
+ abstract String webIdTokenProviderFQCN();
+
+ abstract Integer sessionDurationSecs();
+
+ abstract StsAssumeRoleWithDynamicWebIdentityCredentialsProvider autoBuild(); // not public
+
+ /**
+ * Validates and fully initializes a {@link
+ * StsAssumeRoleWithDynamicWebIdentityCredentialsProvider} instance.
+ *
+ * @return the initialized credentials provider instance.
+ */
+ public StsAssumeRoleWithDynamicWebIdentityCredentialsProvider build() {
+ checkState(audience() != null, "Audience value should not be null");
+ checkState(assumedRoleArn() != null, "The role to assume should not be null");
+ checkState(
+ webIdTokenProviderFQCN() != null,
+ "The web id token provider fully qualified class name should not be null");
+ setCredentialsProviderDelegate(
+ credentialsProviderDelegate()
+ .orElse(
+ CredentialsProviderDelegate.create(
+ Suppliers.memoize(() -> WebIdTokenProvider.create(webIdTokenProviderFQCN())),
+ audience(),
+ assumedRoleArn(),
+ sessionDurationSecs())));
+ return autoBuild();
+ }
+ }
+
+ /**
+ * Given the {@link StsAssumeRoleWithWebIdentityCredentialsProvider} is final and can not be
+ * easily mocked for testing purposes, this simple delegate container will be used to simplify
+ * unit testing.
+ */
+ static class CredentialsProviderDelegate {
+
+ private final Supplier<StsAssumeRoleWithWebIdentityCredentialsProvider>
+ credentialsProviderDelegate;
+
+ CredentialsProviderDelegate(
+ Supplier<StsAssumeRoleWithWebIdentityCredentialsProvider> credentialsProviderDelegate) {
+ this.credentialsProviderDelegate = credentialsProviderDelegate;
+ }
+
+ public static CredentialsProviderDelegate create(
+ Supplier<WebIdTokenProvider> webIdTokenProvider,
+ String audience,
+ String assumedRoleArn,
+ @Nullable Integer sessionDurationSecs) {
+ return new CredentialsProviderDelegate(
+ Suppliers.memoize(
+ () ->
+ createCredentialsDelegate(
+ webIdTokenProvider, audience, assumedRoleArn, sessionDurationSecs)));
+ }
+
+ public AwsCredentials resolveCredentials() {
+ return credentialsProviderDelegate.get().resolveCredentials();
+ }
+
+ public void close() {
+ credentialsProviderDelegate.get().close();
+ }
+
+ static StsAssumeRoleWithWebIdentityCredentialsProvider createCredentialsDelegate(
+ Supplier<WebIdTokenProvider> webIdTokenProvider,
+ String audience,
+ String assumedRoleArn,
+ @Nullable Integer sessionDurationSecs) {
+ return StsAssumeRoleWithWebIdentityCredentialsProvider.builder()
+ .asyncCredentialUpdateEnabled(true)
+ .refreshRequest(
Review Comment:
does this "refreshRequest" correspond to the functionality that
> An implementation of AwsCredentialsProvider that periodically sends an {@link
> AssumeRoleWithWebIdentityRequest} to the AWS
If so why we really need this wrapper provider instead of setup a StsAssumeRoleWithWebIdentityCredentialsProvider with refreshRequest ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2851329560
Would @ahmedabu98 or @chamikaramj or @Abacn be a good reviewer? Or someone else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #34440:
URL: https://github.com/apache/beam/pull/34440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34440:
URL: https://github.com/apache/beam/pull/34440#discussion_r2080730785
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/auth/providers/GoogleADCIdTokenProvider.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.common.auth.providers;
Review Comment:
package name is 9 level nested long. Consider put into .../auth/providers or just under .../auth ?
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/auth/providers/StsAssumeRoleWithDynamicWebIdentityCredentialsProvider.java:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.common.auth.providers;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkState;
+
+import com.fasterxml.jackson.annotation.JsonSubTypes;
+import com.google.auto.value.AutoValue;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.function.Supplier;
+import javax.annotation.Nullable;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Suppliers;
+import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.auth.StsAssumeRoleWithWebIdentityCredentialsProvider;
+import software.amazon.awssdk.services.sts.model.AssumeRoleWithWebIdentityRequest;
+import software.amazon.awssdk.utils.SdkAutoCloseable;
+
+/**
+ * An implementation of AwsCredentialsProvider that periodically sends an {@link
+ * AssumeRoleWithWebIdentityRequest} to the AWS Security Token Service to maintain short-lived
+ * sessions to use for authentication. In particular this class will use a {@link
+ * StsAssumeRoleWithWebIdentityCredentialsProvider} instance as a delegate for the actual
+ * implementation but it takes care of retrieving a refreshed web id token with every credential's
+ * resolution request. This is created using builder().
+ *
+ * @see <a
+ * href="https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html">API
+ * reference</a>
+ */
+@AutoValue
+@JsonSubTypes({
+ @JsonSubTypes.Type(
+ value = AutoValue_StsAssumeRoleWithDynamicWebIdentityCredentialsProvider.class,
+ name = "StsAssumeRoleWithDynamicWebIdentityCredentialsProvider")
+})
+public abstract class StsAssumeRoleWithDynamicWebIdentityCredentialsProvider
Review Comment:
`StsAssumeRoleWithWebIdentityCredentialsProvider` is already very long naming. This wrapper provider is even longer...
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/auth/providers/GoogleADCIdTokenProvider.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.common.auth.providers;
+
+import java.io.IOException;
+import java.util.Arrays;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.GoogleCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.IdTokenCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.IdTokenProvider;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.IdTokenProvider.Option;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * A OIDC web identity token provider implementation that uses the application default credentials
+ * set by the runtime (container, GCE instance, local environment, etc.).
+ */
+public class GoogleADCIdTokenProvider implements WebIdTokenProvider {
Review Comment:
This class is not used. Is it user faced? If so, please elaborate javadoc with instructions and use cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2901016546
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @robertwb for label java.
R: @chamikaramj for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2775597560
Reminder, please take a look at this pr: @m-trieu @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on code in PR #34440:
URL: https://github.com/apache/beam/pull/34440#discussion_r2096352384
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/auth/providers/GoogleADCIdTokenProvider.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.common.auth.providers;
+
+import java.io.IOException;
+import java.util.Arrays;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.GoogleCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.IdTokenCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.IdTokenProvider;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.IdTokenProvider.Option;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * A OIDC web identity token provider implementation that uses the application default credentials
+ * set by the runtime (container, GCE instance, local environment, etc.).
+ */
+public class GoogleADCIdTokenProvider implements WebIdTokenProvider {
Review Comment:
I will add a usage example, basically this class is what enables the STS assume role with WeebId authentication for AWS resources through a configuration (reflection is used to instantiate it) for Dataflow running workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2838588279
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @ahmedabu98 for label java.
R: @chamikaramj for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2756087625
Run Java_Amazon-Web-Services2_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2755669308
fixes #34441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2830260808
Reminder, please take a look at this pr: @Abacn @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2815334736
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @Abacn for label java.
R: @chamikaramj for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2804847079
Reminder, please take a look at this pr: @ahmedabu98 @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #34440:
URL: https://github.com/apache/beam/pull/34440#issuecomment-2892066165
@Abacn thanks for the review!
I will work on the package and class naming recommendations shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on code in PR #34440:
URL: https://github.com/apache/beam/pull/34440#discussion_r2096358395
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/auth/providers/StsAssumeRoleWithDynamicWebIdentityCredentialsProvider.java:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.common.auth.providers;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkState;
+
+import com.fasterxml.jackson.annotation.JsonSubTypes;
+import com.google.auto.value.AutoValue;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.function.Supplier;
+import javax.annotation.Nullable;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Suppliers;
+import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.auth.StsAssumeRoleWithWebIdentityCredentialsProvider;
+import software.amazon.awssdk.services.sts.model.AssumeRoleWithWebIdentityRequest;
+import software.amazon.awssdk.utils.SdkAutoCloseable;
+
+/**
+ * An implementation of AwsCredentialsProvider that periodically sends an {@link
+ * AssumeRoleWithWebIdentityRequest} to the AWS Security Token Service to maintain short-lived
+ * sessions to use for authentication. In particular this class will use a {@link
+ * StsAssumeRoleWithWebIdentityCredentialsProvider} instance as a delegate for the actual
+ * implementation but it takes care of retrieving a refreshed web id token with every credential's
+ * resolution request. This is created using builder().
+ *
+ * @see <a
+ * href="https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html">API
+ * reference</a>
+ */
+@AutoValue
+@JsonSubTypes({
+ @JsonSubTypes.Type(
+ value = AutoValue_StsAssumeRoleWithDynamicWebIdentityCredentialsProvider.class,
+ name = "StsAssumeRoleWithDynamicWebIdentityCredentialsProvider")
+})
+public abstract class StsAssumeRoleWithDynamicWebIdentityCredentialsProvider
+ implements AwsCredentialsProvider, SdkAutoCloseable {
+
+ public static final Integer DEFAULT_SESSION_DURATION_SECS = 3600;
+
+ abstract CredentialsProviderDelegate credentialsProviderDelegate();
+
+ public abstract String audience();
+
+ public abstract String assumedRoleArn();
+
+ public abstract String webIdTokenProviderFQCN();
+
+ @Nullable
+ public abstract Integer sessionDurationSecs();
+
+ @Override
+ public AwsCredentials resolveCredentials() {
+ return credentialsProviderDelegate().resolveCredentials();
+ }
+
+ @Override
+ public void close() {
+ credentialsProviderDelegate().close();
+ }
+
+ /**
+ * Creates a builder for the type {@link StsAssumeRoleWithDynamicWebIdentityCredentialsProvider}.
+ *
+ * @return an initialized builder instance.
+ */
+ public static StsAssumeRoleWithDynamicWebIdentityCredentialsProvider.Builder builder() {
+ return new AutoValue_StsAssumeRoleWithDynamicWebIdentityCredentialsProvider.Builder();
+ }
+
+ /** Builder class for {@link StsAssumeRoleWithDynamicWebIdentityCredentialsProvider}. */
+ @AutoValue.Builder
+ public abstract static class Builder {
+
+ /**
+ * Sets the role to be assumed by the authentication request.
+ *
+ * @param roleArn the AWS role ARN.
+ * @return this builder instance.
+ */
+ public abstract Builder setAssumedRoleArn(String roleArn);
+
+ /**
+ * Sets the audience to be used for the web id token request.
+ *
+ * @param audience the audience value.
+ * @return this builder instance.
+ */
+ public abstract Builder setAudience(String audience);
+
+ /**
+ * The fully qualified class name for the web id token provider. The class should be accessible
+ * in the classpath.
+ *
+ * @param idTokenProviderFQCN the class name.
+ * @return this builder instance.
+ */
+ public abstract Builder setWebIdTokenProviderFQCN(String idTokenProviderFQCN);
+
+ /**
+ * The session duration in seconds for the authentication request, by default this value is
+ * 3600.
+ *
+ * @param durationSecs the duration in seconds.
+ * @return this builder instance.
+ */
+ public abstract Builder setSessionDurationSecs(@Nullable Integer durationSecs);
+
+ /**
+ * For testing purposes, should not be set in a production setup.
+ *
+ * @param delegate The testing delegate to use.
+ * @return this builder instance.
+ */
+ abstract Builder setCredentialsProviderDelegate(CredentialsProviderDelegate delegate);
+
+ abstract Optional<CredentialsProviderDelegate> credentialsProviderDelegate();
+
+ abstract String assumedRoleArn();
+
+ abstract String audience();
+
+ abstract String webIdTokenProviderFQCN();
+
+ abstract Integer sessionDurationSecs();
+
+ abstract StsAssumeRoleWithDynamicWebIdentityCredentialsProvider autoBuild(); // not public
+
+ /**
+ * Validates and fully initializes a {@link
+ * StsAssumeRoleWithDynamicWebIdentityCredentialsProvider} instance.
+ *
+ * @return the initialized credentials provider instance.
+ */
+ public StsAssumeRoleWithDynamicWebIdentityCredentialsProvider build() {
+ checkState(audience() != null, "Audience value should not be null");
+ checkState(assumedRoleArn() != null, "The role to assume should not be null");
+ checkState(
+ webIdTokenProviderFQCN() != null,
+ "The web id token provider fully qualified class name should not be null");
+ setCredentialsProviderDelegate(
+ credentialsProviderDelegate()
+ .orElse(
+ CredentialsProviderDelegate.create(
+ Suppliers.memoize(() -> WebIdTokenProvider.create(webIdTokenProviderFQCN())),
+ audience(),
+ assumedRoleArn(),
+ sessionDurationSecs())));
+ return autoBuild();
+ }
+ }
+
+ /**
+ * Given the {@link StsAssumeRoleWithWebIdentityCredentialsProvider} is final and can not be
+ * easily mocked for testing purposes, this simple delegate container will be used to simplify
+ * unit testing.
+ */
+ static class CredentialsProviderDelegate {
+
+ private final Supplier<StsAssumeRoleWithWebIdentityCredentialsProvider>
+ credentialsProviderDelegate;
+
+ CredentialsProviderDelegate(
+ Supplier<StsAssumeRoleWithWebIdentityCredentialsProvider> credentialsProviderDelegate) {
+ this.credentialsProviderDelegate = credentialsProviderDelegate;
+ }
+
+ public static CredentialsProviderDelegate create(
+ Supplier<WebIdTokenProvider> webIdTokenProvider,
+ String audience,
+ String assumedRoleArn,
+ @Nullable Integer sessionDurationSecs) {
+ return new CredentialsProviderDelegate(
+ Suppliers.memoize(
+ () ->
+ createCredentialsDelegate(
+ webIdTokenProvider, audience, assumedRoleArn, sessionDurationSecs)));
+ }
+
+ public AwsCredentials resolveCredentials() {
+ return credentialsProviderDelegate.get().resolveCredentials();
+ }
+
+ public void close() {
+ credentialsProviderDelegate.get().close();
+ }
+
+ static StsAssumeRoleWithWebIdentityCredentialsProvider createCredentialsDelegate(
+ Supplier<WebIdTokenProvider> webIdTokenProvider,
+ String audience,
+ String assumedRoleArn,
+ @Nullable Integer sessionDurationSecs) {
+ return StsAssumeRoleWithWebIdentityCredentialsProvider.builder()
+ .asyncCredentialUpdateEnabled(true)
+ .refreshRequest(
Review Comment:
See the [configuration serialization](https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsModule.java#L261), it grabs the generated token and serializes it. When the options are propagated to workers the token value is deserialized and used for authentication, this of course works only for a limited period of time (until the provided token value expires).
All classes in the STS package are final, so we need to rely on delegates to implement the logic we need here (which is generate a new token for every refresh request from the AWS auth framework).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enabling long-running jobs to use federated STS assume role authentication for AWS resources. [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on code in PR #34440:
URL: https://github.com/apache/beam/pull/34440#discussion_r2096352384
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/auth/providers/GoogleADCIdTokenProvider.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.common.auth.providers;
+
+import java.io.IOException;
+import java.util.Arrays;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.GoogleCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.IdTokenCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.IdTokenProvider;
+import org.apache.beam.vendor.grpc.v1p69p0.com.google.auth.oauth2.IdTokenProvider.Option;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * A OIDC web identity token provider implementation that uses the application default credentials
+ * set by the runtime (container, GCE instance, local environment, etc.).
+ */
+public class GoogleADCIdTokenProvider implements WebIdTokenProvider {
Review Comment:
I will add a usage example, basically this class is what enables the STS assume role with WeebId authentication for AWS resources through a configuration for Dataflow running workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org