You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/27 18:47:35 UTC
[GitHub] [airflow] o-nikolas commented on a diff in pull request #24670: Update AWS Connection docs and deprecate some extras
o-nikolas commented on code in PR #24670:
URL: https://github.com/apache/airflow/pull/24670#discussion_r907680972
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -51,33 +51,51 @@ Configuring the Connection
Login (optional)
Specify the AWS access key ID used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Password (optional)
Specify the AWS secret access key used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Extra (optional)
Specify the extra parameters (as json dictionary) that can be used in AWS
connection. The following parameters are all optional:
- * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials. You are responsible for renewing these.
-
- * ``role_arn``: If specified, then an *assume_role* will be done to this role.
- * ``aws_account_id``: Used to construct ``role_arn`` if it was not specified.
- * ``aws_iam_role``: Used to construct ``role_arn`` if it was not specified.
- * ``assume_role_kwargs``: Additional ``kwargs`` passed to *assume_role*.
-
+ * ``aws_access_key_id``: AWS access key ID used for the initial connection.
+ * ``aws_secret_access_key``: AWS secret access key used for the initial connection
+ * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials.
+ You are responsible for renewing these.
+ * ``region_name``: AWS Region for the connection.
+ * ``session_kwargs``: Additional **kwargs** passed to
+ `boto3.session.Session <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html>`__.
+ * ``config_kwargs``: Additional **kwargs** used to construct a
+ `botocore.config.Config <https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html>`__
+ passed to `boto3.client <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client>`__
+ and `boto3.resource <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.resource>`__.
+ * ``role_arn``: If specified, then use this role for obtain a set of temporary security credentials will be done to this role.
Review Comment:
```suggestion
* ``role_arn``: If specified, then assume this role, obtaining a set of temporary security credentials using the ``assume_role_method``.
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -51,33 +51,51 @@ Configuring the Connection
Login (optional)
Specify the AWS access key ID used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Password (optional)
Specify the AWS secret access key used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Extra (optional)
Specify the extra parameters (as json dictionary) that can be used in AWS
connection. The following parameters are all optional:
- * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials. You are responsible for renewing these.
-
- * ``role_arn``: If specified, then an *assume_role* will be done to this role.
- * ``aws_account_id``: Used to construct ``role_arn`` if it was not specified.
- * ``aws_iam_role``: Used to construct ``role_arn`` if it was not specified.
- * ``assume_role_kwargs``: Additional ``kwargs`` passed to *assume_role*.
-
+ * ``aws_access_key_id``: AWS access key ID used for the initial connection.
+ * ``aws_secret_access_key``: AWS secret access key used for the initial connection
+ * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials.
+ You are responsible for renewing these.
+ * ``region_name``: AWS Region for the connection.
+ * ``session_kwargs``: Additional **kwargs** passed to
+ `boto3.session.Session <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html>`__.
+ * ``config_kwargs``: Additional **kwargs** used to construct a
+ `botocore.config.Config <https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html>`__
+ passed to `boto3.client <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client>`__
+ and `boto3.resource <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.resource>`__.
+ * ``role_arn``: If specified, then use this role for obtain a set of temporary security credentials will be done to this role.
+ * ``assume_role_method``: `AWS STS Client method <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#client>`__
+ for `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__,
+ if not specified than `assume_role <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role>`__ use.
+ * ``assume_role_kwargs``: Additional **kwargs** passed to ``assume_role_method``.
* ``host``: Endpoint URL for the connection.
- * ``region_name``: AWS region for the connection.
- * ``external_id``: AWS external ID for the connection (deprecated, rather use ``assume_role_kwargs``).
+ * ``s3_config_file``: Path to local credentials file.
+ * ``s3_config_format``: ``s3_config_file`` format, one of
+ `aws <https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-settings>`_,
+ `boto <http://boto.cloudhackers.com/en/latest/boto_config_tut.html#details>`_ or
+ `s3cmd <https://s3tools.org/kb/item14.htm>`_ if not specified than **boto** use.
+ * ``profile``: If you are getting your credentials from the ``s3_config_file``
+ you can specify the profile with this parameter.
- * ``config_kwargs``: Additional ``kwargs`` used to construct a ``botocore.config.Config`` passed to *boto3.client* and *boto3.resource*.
- * ``session_kwargs``: Additional ``kwargs`` passed to *boto3.session.Session*.
+.. warning:: Extra parameters below deprecated and will remove in future version of provider.
Review Comment:
Also I'd add the s3 related ones above down here.
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -51,33 +51,51 @@ Configuring the Connection
Login (optional)
Specify the AWS access key ID used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Password (optional)
Specify the AWS secret access key used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Extra (optional)
Specify the extra parameters (as json dictionary) that can be used in AWS
connection. The following parameters are all optional:
- * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials. You are responsible for renewing these.
-
- * ``role_arn``: If specified, then an *assume_role* will be done to this role.
- * ``aws_account_id``: Used to construct ``role_arn`` if it was not specified.
- * ``aws_iam_role``: Used to construct ``role_arn`` if it was not specified.
- * ``assume_role_kwargs``: Additional ``kwargs`` passed to *assume_role*.
-
+ * ``aws_access_key_id``: AWS access key ID used for the initial connection.
+ * ``aws_secret_access_key``: AWS secret access key used for the initial connection
+ * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials.
+ You are responsible for renewing these.
+ * ``region_name``: AWS Region for the connection.
+ * ``session_kwargs``: Additional **kwargs** passed to
+ `boto3.session.Session <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html>`__.
+ * ``config_kwargs``: Additional **kwargs** used to construct a
+ `botocore.config.Config <https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html>`__
+ passed to `boto3.client <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client>`__
+ and `boto3.resource <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.resource>`__.
+ * ``role_arn``: If specified, then use this role for obtain a set of temporary security credentials will be done to this role.
+ * ``assume_role_method``: `AWS STS Client method <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#client>`__
+ for `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__,
+ if not specified than `assume_role <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role>`__ use.
+ * ``assume_role_kwargs``: Additional **kwargs** passed to ``assume_role_method``.
* ``host``: Endpoint URL for the connection.
- * ``region_name``: AWS region for the connection.
- * ``external_id``: AWS external ID for the connection (deprecated, rather use ``assume_role_kwargs``).
+ * ``s3_config_file``: Path to local credentials file.
+ * ``s3_config_format``: ``s3_config_file`` format, one of
Review Comment:
If I read your descriptions of this PR right, then I think we agree that these shouldn't be here. I wouldn't add documentation for it now if we're planning on removing it and don't want people to use it. At the very least, note here that these are deprecated.
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -51,33 +51,51 @@ Configuring the Connection
Login (optional)
Specify the AWS access key ID used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Password (optional)
Specify the AWS secret access key used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Extra (optional)
Specify the extra parameters (as json dictionary) that can be used in AWS
connection. The following parameters are all optional:
- * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials. You are responsible for renewing these.
-
- * ``role_arn``: If specified, then an *assume_role* will be done to this role.
- * ``aws_account_id``: Used to construct ``role_arn`` if it was not specified.
- * ``aws_iam_role``: Used to construct ``role_arn`` if it was not specified.
- * ``assume_role_kwargs``: Additional ``kwargs`` passed to *assume_role*.
-
+ * ``aws_access_key_id``: AWS access key ID used for the initial connection.
+ * ``aws_secret_access_key``: AWS secret access key used for the initial connection
+ * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials.
+ You are responsible for renewing these.
+ * ``region_name``: AWS Region for the connection.
+ * ``session_kwargs``: Additional **kwargs** passed to
+ `boto3.session.Session <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html>`__.
+ * ``config_kwargs``: Additional **kwargs** used to construct a
+ `botocore.config.Config <https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html>`__
+ passed to `boto3.client <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client>`__
+ and `boto3.resource <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.resource>`__.
+ * ``role_arn``: If specified, then use this role for obtain a set of temporary security credentials will be done to this role.
+ * ``assume_role_method``: `AWS STS Client method <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#client>`__
+ for `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__,
+ if not specified than `assume_role <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role>`__ use.
+ * ``assume_role_kwargs``: Additional **kwargs** passed to ``assume_role_method``.
* ``host``: Endpoint URL for the connection.
- * ``region_name``: AWS region for the connection.
- * ``external_id``: AWS external ID for the connection (deprecated, rather use ``assume_role_kwargs``).
+ * ``s3_config_file``: Path to local credentials file.
+ * ``s3_config_format``: ``s3_config_file`` format, one of
+ `aws <https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-settings>`_,
+ `boto <http://boto.cloudhackers.com/en/latest/boto_config_tut.html#details>`_ or
+ `s3cmd <https://s3tools.org/kb/item14.htm>`_ if not specified than **boto** use.
Review Comment:
```suggestion
`s3cmd <https://s3tools.org/kb/item14.htm>`_ if not specified then **boto** is used.
```
##########
airflow/providers/amazon/aws/hooks/base_aws.py:
##########
@@ -83,6 +83,15 @@ def create_session(self) -> boto3.session.Session:
self.extra_config["session_kwargs"],
)
session_kwargs = self.extra_config["session_kwargs"]
+
+ if "profile" in self.extra_config and "s3_config_file" not in self.extra_config:
Review Comment:
Why is s3 specific logic being added to the AWS base session class?
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -51,33 +51,51 @@ Configuring the Connection
Login (optional)
Specify the AWS access key ID used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Password (optional)
Specify the AWS secret access key used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Extra (optional)
Specify the extra parameters (as json dictionary) that can be used in AWS
connection. The following parameters are all optional:
- * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials. You are responsible for renewing these.
-
- * ``role_arn``: If specified, then an *assume_role* will be done to this role.
- * ``aws_account_id``: Used to construct ``role_arn`` if it was not specified.
- * ``aws_iam_role``: Used to construct ``role_arn`` if it was not specified.
- * ``assume_role_kwargs``: Additional ``kwargs`` passed to *assume_role*.
-
+ * ``aws_access_key_id``: AWS access key ID used for the initial connection.
+ * ``aws_secret_access_key``: AWS secret access key used for the initial connection
+ * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials.
+ You are responsible for renewing these.
+ * ``region_name``: AWS Region for the connection.
+ * ``session_kwargs``: Additional **kwargs** passed to
+ `boto3.session.Session <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html>`__.
+ * ``config_kwargs``: Additional **kwargs** used to construct a
+ `botocore.config.Config <https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html>`__
+ passed to `boto3.client <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client>`__
+ and `boto3.resource <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.resource>`__.
+ * ``role_arn``: If specified, then use this role for obtain a set of temporary security credentials will be done to this role.
+ * ``assume_role_method``: `AWS STS Client method <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#client>`__
+ for `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__,
+ if not specified than `assume_role <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role>`__ use.
Review Comment:
I find this section a bit confusing the way it's written now, maybe just list the acceptable values with a short description and link to the docs for each:
1. `assume_role_with_web_identity` ([doc link](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html))
1. `assume_role` ([doc link](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html))
1. `assume_role_with_saml` ([doc link](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithSAML.html))
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -51,33 +51,51 @@ Configuring the Connection
Login (optional)
Specify the AWS access key ID used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Password (optional)
Specify the AWS secret access key used for the initial connection.
- If you do an *assume_role* by specifying a ``role_arn`` in the **Extra** field,
+ If you do an `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__
+ by specifying a ``role_arn`` in the **Extra** field,
then temporary credentials will be used for subsequent calls to AWS.
Extra (optional)
Specify the extra parameters (as json dictionary) that can be used in AWS
connection. The following parameters are all optional:
- * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials. You are responsible for renewing these.
-
- * ``role_arn``: If specified, then an *assume_role* will be done to this role.
- * ``aws_account_id``: Used to construct ``role_arn`` if it was not specified.
- * ``aws_iam_role``: Used to construct ``role_arn`` if it was not specified.
- * ``assume_role_kwargs``: Additional ``kwargs`` passed to *assume_role*.
-
+ * ``aws_access_key_id``: AWS access key ID used for the initial connection.
+ * ``aws_secret_access_key``: AWS secret access key used for the initial connection
+ * ``aws_session_token``: AWS session token used for the initial connection if you use external credentials.
+ You are responsible for renewing these.
+ * ``region_name``: AWS Region for the connection.
+ * ``session_kwargs``: Additional **kwargs** passed to
+ `boto3.session.Session <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html>`__.
+ * ``config_kwargs``: Additional **kwargs** used to construct a
+ `botocore.config.Config <https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html>`__
+ passed to `boto3.client <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client>`__
+ and `boto3.resource <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.resource>`__.
+ * ``role_arn``: If specified, then use this role for obtain a set of temporary security credentials will be done to this role.
+ * ``assume_role_method``: `AWS STS Client method <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#client>`__
+ for `assume role <https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html>`__,
+ if not specified than `assume_role <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role>`__ use.
+ * ``assume_role_kwargs``: Additional **kwargs** passed to ``assume_role_method``.
* ``host``: Endpoint URL for the connection.
- * ``region_name``: AWS region for the connection.
- * ``external_id``: AWS external ID for the connection (deprecated, rather use ``assume_role_kwargs``).
+ * ``s3_config_file``: Path to local credentials file.
+ * ``s3_config_format``: ``s3_config_file`` format, one of
+ `aws <https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-settings>`_,
+ `boto <http://boto.cloudhackers.com/en/latest/boto_config_tut.html#details>`_ or
+ `s3cmd <https://s3tools.org/kb/item14.htm>`_ if not specified than **boto** use.
+ * ``profile``: If you are getting your credentials from the ``s3_config_file``
+ you can specify the profile with this parameter.
- * ``config_kwargs``: Additional ``kwargs`` used to construct a ``botocore.config.Config`` passed to *boto3.client* and *boto3.resource*.
- * ``session_kwargs``: Additional ``kwargs`` passed to *boto3.session.Session*.
+.. warning:: Extra parameters below deprecated and will remove in future version of provider.
Review Comment:
```suggestion
.. warning:: Extra parameters below are deprecated and will be removed in a future version of this provider.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org