You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pgagnon (via GitHub)" <gi...@apache.org> on 2023/02/20 05:07:39 UTC
[GitHub] [airflow] pgagnon opened a new pull request, #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
pgagnon opened a new pull request, #29623:
URL: https://github.com/apache/airflow/pull/29623
This PR implements a new `web_identity_token_loader` for the AWS connection `AssumeRoleWithWebIdentity` authentication mechanism that fetches the access token from a location on the Airflow instance's filesystem.
This can be especially useful for Kubernetes environments with the [Service Account Volume Projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection) feature enabled.
This is similar to the built-in boto mechanism configured through the `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE` environment variables but is much more flexible as it can be configured individually at the Airflow connection level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "pgagnon (via GitHub)" <gi...@apache.org>.
pgagnon commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112140747
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -311,20 +311,32 @@ def _get_web_identity_credential_fetcher(
) -> botocore.credentials.AssumeRoleWithWebIdentityCredentialFetcher:
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
- federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+ federation = str(self.extra_config.get("assume_role_with_web_identity_federation"))
+
+ web_identity_token_loader = {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation, lambda: None)()
+
+ if not web_identity_token_loader:
Review Comment:
It will because the callable is called right before assignment:
```
$ python
Python 3.9.7 (default, Dec 5 2021, 13:21:59)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def _get_file_token_loader():
... return "file"
...
>>> def _get_google_identity_token_loader():
... return "google"
...
>>> federation = "test"
>>>
>>> web_identity_token_loader = {
... "file": _get_file_token_loader,
... "google": _get_google_identity_token_loader,
... }.get(federation, lambda: None)()
>>>
>>> print("hit" if not web_identity_token_loader else "not hit")
hit
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "pgagnon (via GitHub)" <gi...@apache.org>.
pgagnon commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112391709
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
> Seems like it provide ability to set different web identity token file, isn't it? 🤔
Yes; I don't see a reason not to allow parameterization here. In fact, I think not doing so would be inconsistent and counter to the goal of allowing the feature to be fully configured through the connections subsystem.
> I guess this option exists for allow create own method auth to AWS without overwrite everything in provider. If it complicated better to think how it make easier to end users.
Custom classes will always only be used by our more sophisticated users. I do believe that they have their place in Airflow, but they shouldn't be considered a replacement for more accessible APIs, even if they functionally allow the same implementation.
> Airflow [`AssumeRole`](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) in two steps
>
> 1. Obtain initial session by: either direct credentials or use `boto3` default strategy for obtain credentials (Environment Variables, EC2 IAM Profile, ECS Execution Role and etc)
> 2. Use session obtained in Step 1 for assume role thought STS API [`AssumeRole`](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
Unfortunately, this doesn't work out of the box for the vast majority of operators. Furthermore, this doesn't address the use case; there are many ways to obtain temporary credentials, but none currently allow configuring `AssumeRoleWithWebIdentity` without relying on external configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #29623:
URL: https://github.com/apache/airflow/pull/29623#issuecomment-1448725576
> @ashb you showed great disrespect with your actions:
>
> * remove change request of other member
>
> * resolve unresolved discussion
>
> * avoid discussion
>
> * merge potential insecure feature
I'm sorry but no. Just no. You do not get to play the "potential insecure feature". This is categorical more secure as it gives users an option to avoid having to bake key/secret in to connections. Are you suggesting that we don't allow putting aws key id and secret in connections, because that sure as hell is no less secure than this.
Avoid discussion? You're kidding right. There was a lengthy discussion, and from my side you flatly refused to believe any use case than your own, and didn't acknowledge that there are different users with abilities and knowledge of AWS _and this helps them_.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "pgagnon (via GitHub)" <gi...@apache.org>.
pgagnon commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1111979656
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
Yes, that's also an option and can be preferred in many cases, but this explicitly allows configuration through the Airflow connections subsystem only.
The same can be said about almost all connection types, but we enable specific configurations through extras to allow flexibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #29623:
URL: https://github.com/apache/airflow/pull/29623#issuecomment-1448695559
@ashb you showed great disrespect with your actions:
- remove change request of other member
- resolve unresolved discussion
- avoid discussion
- merge potential insecure feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1113247672
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
And as Philippe said, botocore is _not_ a private module.
Importing a public class from `botocore.credentials` is 100% not a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112397450
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
> Unfortunately, this doesn't work out of the box for the vast majority of operators. Furthermore, this doesn't address the use case; there are many ways to obtain temporary credentials, but none currently allow configuring AssumeRoleWithWebIdentity without relying on external configs.
It would work with all operators which required AWS Connection, the DAG I provide as a sample, in prod all you need is:
1. Setup in your environment `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`
2. Allow for `AWS_ROLE_ARN` to assume another role. This could be setup thought AWS IAM which do not required to change anything in Airflow environment.
3. Setup your connection which in extra `{"role_arn": "your-required-role-here"}`
Repeat step 2 and 3 for new roles what you required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112132611
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -311,20 +311,32 @@ def _get_web_identity_credential_fetcher(
) -> botocore.credentials.AssumeRoleWithWebIdentityCredentialFetcher:
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
- federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+ federation = str(self.extra_config.get("assume_role_with_web_identity_federation"))
+
+ web_identity_token_loader = {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation, lambda: None)()
+
+ if not web_identity_token_loader:
Review Comment:
Since you have a default of `lambda: None` this is never going to be hit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1114381838
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
> The point of this feature is to allow different Airflow connections to have different styles of credentials. I'm marking this point as resolved now as there are no private APIs/attrs in use.
My point initially that this changes not required at all, because everything is available right now in Amazon Provider.
Make users allow to setup something which not available thought `botocore.session.Session` (there is no way to change env var here or setup default values) or `botocore.config.Config` could be potentially unsafe and would make administrators who maintain Airflow unhappy.
AWS do not provide public interface for iterate with this function directly, more probably for security reason, but we would not allow to make in community provider). Only public interface for that potentially "unsafe" capabilities thought profiles, AWS Config file and ENV VARs which under administrators control, another option it is `aws-cli` but this not option in this case.
I can't find any use cases for this feature, as well as AWS do not even provide any documentation for that, rather than setup environment variables for k8s pod.
Right now users and administrators have everything for do different and potentially unsafe stuff when it comes to credentials to AWS.
1. Provide Environment Variables, and config files. This part under control of administrators
2. Write own SessionFactory, yeah AWS provider is unique provider which allow overwrite logic for obtain
3. Obtain initial session and assume to specific role. The same, this is under control of administrators, default session obtained thought boto3 credential strategy (p1) and other stuff happen thought AWS IAM, permission could be revoked and monitored.
```json
{
"role_arn": "arn:aws:iam::112223334444:role/my_role",
"region_name": "ap-southeast-2"
}
```
This PR it is nice way to shot in foot or cast security wormhole, we should avoid to use any files/classes which passed thought connection. The only one way to allow this to the community provider:
1. Make it optional
2. Enable thought Airflow Configurations `[aws] allow_unsafe`
3. Turned off this functional by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1119829038
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
How come it help? Could you provide more detail and samples?
What is wrong with general approach which also might obtain credentials without persistent credentials?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112131738
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
Makes sense to me. WDYT @Taragolis ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112313305
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
I don't want to be a "party breaker" but I have to. 🤣
It is not even about current implementation because it could be improved by remove some hacks and private things usage.
More about that right `BaseSessionFactory` it contains a lot of hacks and private method usage and unfortunetly we could check only part of this hacks, I'm not even sure that that SAML over SPNEGO and Federation over Google still work because everything just mocking, unfortunetly the same situation here. Ideally we need some kind of System Tests not only for Operators but also for Connections 😿
And there is still exists alternatives:
1. Profile from AWS Config File. I think if user could provide multiple tokens per POD it is also possible provide additional file with config
2. Build own class over [`BaseSessionFactory`](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#session-factory)
3. Obtain initial credentials by [default `boto3` credentials strategy](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) (in this case Environment Variables) and assume to specific role, of course it required additional IAM configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1113244622
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
Speaking to strictly a user feature point of view: Supporting this feature in Airflow connections that work with any AWS operator is a huge win. So the DAG approach you have right now isn't really viable, not for users who aren't proficient in Python -- which lets not forget is many Airflow users.
So lets state my ground truth: Being able to configure an AWS connection exclusively through the Airflow UI to use a web identity token, that can then be used with all existing AWS operators, is a good feature and one we will accept.
We can talk about implementation details, but "just write a python operator do to it" is not an answer to this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "pgagnon (via GitHub)" <gi...@apache.org>.
pgagnon commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112403111
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
It still doesn't address the setting where you can't set external configs. The objective is to allow configuring connections that use `AssumeRoleWithWebIdentity` without relying on environment variables and AWS's configuration/shared credentials file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1113233527
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -311,20 +311,32 @@ def _get_web_identity_credential_fetcher(
) -> botocore.credentials.AssumeRoleWithWebIdentityCredentialFetcher:
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
- federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+ federation = str(self.extra_config.get("assume_role_with_web_identity_federation"))
+
+ web_identity_token_loader = {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation, lambda: None)()
+
+ if not web_identity_token_loader:
Review Comment:
Oh missed the `()` gotcha.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1119802877
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -311,20 +311,32 @@ def _get_web_identity_credential_fetcher(
) -> botocore.credentials.AssumeRoleWithWebIdentityCredentialFetcher:
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
- federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+ federation = str(self.extra_config.get("assume_role_with_web_identity_federation"))
+
+ web_identity_token_loader = {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation, lambda: None)()
+
+ if not web_identity_token_loader:
+ raise AirflowException(f"Unsupported federation: {federation}.")
+
return botocore.credentials.AssumeRoleWithWebIdentityCredentialFetcher(
client_creator=client_creator,
web_identity_token_loader=web_identity_token_loader,
Review Comment:
This "default" lambda pattern is a bit odd/unusual, so lets do this instead
```suggestion
web_identity_token_loader = {
"file": self._get_file_token_loader,
"google": self._get_google_identity_token_loader,
}.get(federation)
if not web_identity_token_loader:
raise AirflowException(f"Unsupported federation: {federation}.")
return botocore.credentials.AssumeRoleWithWebIdentityCredentialFetcher(
client_creator=client_creator,
web_identity_token_loader=web_identity_token_loader(),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #29623:
URL: https://github.com/apache/airflow/pull/29623#issuecomment-1448728749
This is designed to be used by feature such as https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery where the kubelet is responsible for managing the contents of this web identity file.
This whole PR lets that be usable and means that users don't have to bake credentials in to the image/pod or even in connections -- i.e. they make use of short-lived credentials --> are more secure than static key pairs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112407307
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
Use non-documented features of `botocore` and private components 🙄 What could definitely goes wrong?
I know at least `aiobotocore`, `PynamoDB` had this problems in the past (or even nowadays) and I wondering why Airflow still do not have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "pgagnon (via GitHub)" <gi...@apache.org>.
pgagnon commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112410526
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
`botocore` is a low-level API, but it is not private. I'd be _very_ happy to work with you toward addressing your concerns about the implementation (really).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1111561157
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
I guess we do not need to implement anything, everything could be done through AWS Config / AWS Shared Credential Files and profiles, see available options:
- https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-a-configuration-file
- https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-settings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1116025807
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
> I do not find this argument compelling at all. If a user was so inclined they could simply obtain an access key id, secret and session token by calling AssumeRoleWIthWebIdentity with the STS API endpoint directly. I fail to see how this raises any security concerns.
Yep what you write is is controlled by IAM permissions and required initial boto3/botocore session and STS.Client.
Your proposal more about how obtain this initial session by provide custom filepath and role-arn from the connection, which can not be limited by administrators. And in situation if one pod have multiple web identity token files mounted user might easily change default AWS SDK behaviour and get information different rather than initially specified for `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`.
I want to highlight that use this env variables it is expected integration according to AWS docs, I can't find any other ways:
- https://docs.aws.amazon.com/eks/latest/userguide/pod-configuration.html
- https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
If we look throught the `botocore` code, there is no way to legit redefine this behaivior, rather than use profiles.
And just wondering what is wrong with general approach?
pod has `AWS_ROLE_ARN` for example `arn:aws:iam::111122223333:role/my-role` and `AWS_WEB_IDENTITY_TOKEN_FILE` for example `/var/run/secrets/eks.amazonaws.com/serviceaccount/token`, in this case AWS Connection obtain initial session with role `arn:aws:iam::111122223333:role/my-role` and if it required to switch to another like `arn:aws:iam::5555666677778888:role/my-role` than you/administrator need to allow through AWS IAM access to assume `arn:aws:iam::5555666677778888:role/my-another-role` from `arn:aws:iam::111122223333:role/my-role`. As soon as you have this everything you only need to specify in connection Extra `{"role_arn": "arn:aws:iam::5555666677778888:role/my-another-role"}` and **do not specify** `aws_access_key_id`, `aws_secret_access_key`, `aws_session_token`, `profile_name`.
This how it works in the other cases when need to switch role, e.g. in EC2 Instance with attached IAM profile or ECS Task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "pgagnon (via GitHub)" <gi...@apache.org>.
pgagnon commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1120025591
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -311,20 +311,32 @@ def _get_web_identity_credential_fetcher(
) -> botocore.credentials.AssumeRoleWithWebIdentityCredentialFetcher:
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
- federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+ federation = str(self.extra_config.get("assume_role_with_web_identity_federation"))
+
+ web_identity_token_loader = {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation, lambda: None)()
+
+ if not web_identity_token_loader:
+ raise AirflowException(f"Unsupported federation: {federation}.")
+
return botocore.credentials.AssumeRoleWithWebIdentityCredentialFetcher(
client_creator=client_creator,
web_identity_token_loader=web_identity_token_loader,
Review Comment:
Agreed, this is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1114196396
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
> ```python
> token_file = self.extra_config.get("assume_role_with_web_identity_token_file") or os.getenv(
> AssumeRoleWithWebIdentityProvider._CONFIG_TO_ENV_VAR["web_identity_token_file"]
> )
> ```
Right, that was easy enough to fix -- and a concrete review pointing at the specific instance would have been more helpful than talking in generics.
The point of this feature is to allow different Airflow connections to have different styles of credentials. I'm marking this point as resolved now as there are no private APIs/attrs in use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "pgagnon (via GitHub)" <gi...@apache.org>.
pgagnon commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1114427816
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
> AWS do not provide public interface for iterate with this function directly, more probably for security reason
I do not find this argument compelling at all. If a user was so inclined they could simply obtain an access key id, secret and session token by calling AssumeRoleWIthWebIdentity with the STS API endpoint directly. I fail to see how this raises any security concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "pgagnon (via GitHub)" <gi...@apache.org>.
pgagnon commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112360505
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
@Taragolis I think two areas are being raised here. On the implementation, I am well aware that it can be improved, and I fully expected to go through a few rounds to get this in shape. If you have specific review comments, I would happily work with you on this. 🙂
Regarding the alternatives that you have outlined, I have considered them previously, and I do not think they address the objective. Specifically:
1. The objective is more about using multiple `role_arn` than multiple tokens. It is widespread in practice for users not to have the ability to override config files (for instance, in corporate environments or on PaaS), so defining profiles in the AWS config or shared credentials file is not always practical.
2. This shares the same concern as point 1 regarding the inability to overwrite config files. Also, I think this option would be dismissed as too complex and burdensome by 99% of Airflow users.
3. This does not permit configuration through the Airflow connections subsystem without relying on external configs, which are (1) not necessarily mutable and (2) more-or-less static. This would be possible if `boto3` allowed passing `web_identity_token_file,` `role_arn`, or `role_session_name` as parameters to clients or sessions, but sadly it does not at this point.
This being said, I share your concern about the lack of system tests for connections, and this is something worthwhile to look at but probably out of the scope of this discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112384223
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
> The objective is more about using multiple role_arn than multiple tokens. It is widespread in practice for users not to have the ability to override config files (for instance, in corporate environments or on PaaS), so defining profiles in the AWS config or shared credentials file is not always practical.
Seems like it provide ability to set different web identity token file, isn't it? 🤔
```python
token_file = self.extra_config.get("assume_role_with_web_identity_token_file") or os.getenv(
AssumeRoleWithWebIdentityProvider._CONFIG_TO_ENV_VAR["web_identity_token_file"]
)
```
> This shares the same concern as point 1 regarding the inability to overwrite config files. Also, I think this option would be dismissed as too complex and burdensome by 99% of Airflow users.
I guess this option exists for allow create own method auth to AWS without overwrite everything in provider. If it complicated better to think how it make easier to end users.
> This does not permit configuration through the Airflow connections subsystem without relying on external configs, which are (1) not necessarily mutable and (2) more-or-less static.
Airflow [`AssumeRole`](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) in two steps
1. Obtain initial session by: either direct credentials or use `boto3` default strategy for obtain credentials (Environment Variables, EC2 IAM Profile, ECS Execution Role and etc)
2. Use session obtained in Step 1 for assume role thought STS API [`AssumeRole`](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
So I can't see any problem for obtain in EKS IRSA or by other SA (`AWS_ROLE_ARN`, `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_SESSION_NAME` environment variables) and after that assume to required role.
That simple snippet DAG for check
```python
import os
from airflow import DAG
from airflow.decorators import task
from airflow.models import Connection
import pendulum
with DAG(
dag_id="test-irsa-role",
start_date=pendulum.datetime(2023, 2, 1, tz="UTC"),
schedule=None,
catchup=False,
tags=["aws", "assume-role", "29623"]
):
@task
def check_connection():
os.environ["AWS_ROLE_ARN"] = "arn:aws:iam::111111111111:role/spam-egg-role"
os.environ["AWS_WEB_IDENTITY_TOKEN_FILE"] = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
conn_id = "fake-conn-for-test"
region_name = "eu-west-1"
assumed_role = "arn:aws:iam::111111111111:role/foo-bar-role"
assert assumed_role != os.environ["AWS_ROLE_ARN"]
conn = Connection(
conn_id=conn_id,
conn_type="aws",
extra={
"role_arn": assumed_role,
"region_name": region_name
}
)
env_key = f"AIRFLOW_CONN_{conn.conn_id.upper()}"
os.environ[env_key] = conn.get_uri()
result, message = conn.test_connection()
if result:
return message
raise PermissionError(message)
check_connection()
```
> This would be possible if boto3 allowed passing web_identity_token_file, role_arn, or role_session_name as parameters to clients or sessions, but sadly it does not at this point.
This would be nice to address to [boto team](https://github.com/boto), and may be they could share why it is not implemented into their SDK, I guess some reason exists for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "pgagnon (via GitHub)" <gi...@apache.org>.
pgagnon commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112403111
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
It still doesn't address the setting where you can't set external configs. The objective is to allow configuring connections that use `AssumeRoleWithWebIdentity` without relying on environment variables and AWS's configuration/shared credentials file.
edit: (btw thank you for clarifying, I did misread your code snippet earlier.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112397450
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
> Unfortunately, this doesn't work out of the box for the vast majority of operators. Furthermore, this doesn't address the use case; there are many ways to obtain temporary credentials, but none currently allow configuring AssumeRoleWithWebIdentity without relying on external configs.
It would work with all operators which required AWS Connection, the DAG I provide as a sample, in prod all you need is:
1. Setup in your environment `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`
2. Allow for `AWS_ROLE_ARN` to assume another role. This could be setup thought AWS IAM which do not required to change anything in Airflow environment.
3. Setup your connection which in extra {"role_arn": "your-required-role-here"}
Repeat step 2 and 3 for new roles what you required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112397450
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
> Unfortunately, this doesn't work out of the box for the vast majority of operators. Furthermore, this doesn't address the use case; there are many ways to obtain temporary credentials, but none currently allow configuring AssumeRoleWithWebIdentity without relying on external configs.
It would work with all operators which required AWS Connection, the DAG I provide as a sample, in prod all you need:
1. Setup in your environment `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`
2. Allow for `AWS_ROLE_ARN` to assume another role. This could be setup thought AWS IAM which do not required to change anything in Airflow environment.
3. Setup your connection which in extra `{"role_arn": "your-required-role-here"}`
Repeat step 2 and 3 for new roles what you required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1113290164
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
>Speaking to strictly a user feature point of view: Supporting this feature in Airflow connections that work with any AWS operator is a huge win.
@ashb Let me repeat one more time: "This **would** work right now without custom code in DAG, **>>>> DAG only for sample <<<<** ), the same as [snippet as provided in documentation for check](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#snippet-to-create-connection-and-convert-to-uri).
There is exists option which tested and covered:
- Step 1: Obtain by `boto3` default credentials strategy
- Step 2: Assume required role
This work thought **single connection**, the same way as it work when Airflow deployed in EC2 Instances or ECS Tasks.
And in opposite this solution (as well as SPHEGO and Google Federation) mock everything, we even do not know if `botocore` change this part because we mock everything.
> And as Philippe said, botocore is not a private module.
>
> Importing a public class from botocore.credentials is 100% not a problem.
Huh 🤔?
```python
token_file = self.extra_config.get("assume_role_with_web_identity_token_file") or os.getenv(
AssumeRoleWithWebIdentityProvider._CONFIG_TO_ENV_VAR["web_identity_token_file"]
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "pgagnon (via GitHub)" <gi...@apache.org>.
pgagnon commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1112002541
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
Just to expand, if boto allowed passing `web_identity_token_file`, `role_arn`, and `role_session_name` as kwargs, then I would agree with you that nothing needs to be implemented, but AFAIK, that is not the case; the only options available are (1) environment variables and (2) aws configuration/shared credentials file, both of which are external and more or less static configuration mechanisms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #29623:
URL: https://github.com/apache/airflow/pull/29623#discussion_r1119800410
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher(
base_session = self.basic_session._session or botocore.session.get_session()
client_creator = base_session.create_client
federation = self.extra_config.get("assume_role_with_web_identity_federation")
- if federation == "google":
- web_identity_token_loader = self._get_google_identity_token_loader()
- else:
- raise AirflowException(
- f'Unsupported federation: {federation}. Currently "google" only are supported.'
- )
+
+ web_identity_token_loader = (
+ {
+ "file": self._get_file_token_loader,
+ "google": self._get_google_identity_token_loader,
+ }.get(federation)()
+ if type(federation) == str
+ else None
+ )
Review Comment:
The point of this feature is for cases such as where a user deploys with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars, but wants to move towards a more secure approach.
The long and the short of it: This is useful to help users use AWS in the many different ways they want to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb merged PR #29623:
URL: https://github.com/apache/airflow/pull/29623
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org