You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Taragolis (via GitHub)" <gi...@apache.org> on 2023/02/02 20:06:21 UTC
[GitHub] [airflow] Taragolis opened a new pull request, #29341: Add clarification about AWS Connection behaviour
Taragolis opened a new pull request, #29341:
URL: https://github.com/apache/airflow/pull/29341
It is not uncommon when user tried to use Amazon provider for interact with AWS API compatible services (MinIO or others).
Even if we do not officially support them better provide additional information how to correctly understand AWS Connection testing.
In additional I add information about missing connections and deprecated AwsBaseHook behaviour.
Feel free to suggest any improvements in text
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on a diff in pull request #29341: Add clarification about AWS Connection behaviour
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #29341:
URL: https://github.com/apache/airflow/pull/29341#discussion_r1095054455
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -20,8 +20,21 @@
Amazon Web Services Connection
==============================
-The Amazon Web Services connection type enables the :ref:`AWS Integrations
-<AWS>`.
+The Amazon Web Services connection type enables the :ref:`AWS Integrations <AWS>`.
+
+.. important:: Amazon Web Services Connection could be tested in the UI/API or by call
Review Comment:
```suggestion
.. important:: The Amazon Web Services Connection can be tested in the UI/API or by calling
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -20,8 +20,21 @@
Amazon Web Services Connection
==============================
-The Amazon Web Services connection type enables the :ref:`AWS Integrations
-<AWS>`.
+The Amazon Web Services connection type enables the :ref:`AWS Integrations <AWS>`.
+
+.. important:: Amazon Web Services Connection could be tested in the UI/API or by call
+ :meth:`~airflow.providers.amazon.aws.hooks.base_aws.AwsGenericHook.test_connection`,
+ it is **important** to correct interpret result of this test.
Review Comment:
```suggestion
it is **important** to correctly interpret the result of this test.
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -32,19 +45,26 @@ Alternatively, one can pass credentials in as a Connection initialisation parame
To use IAM instance profile, create an "empty" connection (i.e. one with no AWS Access Key ID or AWS Secret Access Key
specified, or ``aws://``).
-
Default Connection IDs
-----------------------
The default connection ID is ``aws_default``. If the environment/machine where you are running Airflow has the
file credentials in ``/home/.aws/``, and the default connection has user and pass fields empty, it will take
automatically the credentials from there.
-.. note:: Previously, the ``aws_default`` connection had the "extras" field set to ``{"region_name": "us-east-1"}``
+.. important:: Previously, the ``aws_default`` connection had the "extras" field set to ``{"region_name": "us-east-1"}``
on install. This means that by default the ``aws_default`` connection used the ``us-east-1`` region.
This is no longer the case and the region needs to be set manually, either in the connection screens in Airflow,
or via the ``AWS_DEFAULT_REGION`` environment variable.
+.. caution:: If you do not set ``[database] load_default_connections`` to ``True``
+ most probably you do not have ``aws_default``. By historical reason Amazon Provider
Review Comment:
```suggestion
most probably you do not have ``aws_default``. For historical reasons, the Amazon Provider
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -32,19 +45,26 @@ Alternatively, one can pass credentials in as a Connection initialisation parame
To use IAM instance profile, create an "empty" connection (i.e. one with no AWS Access Key ID or AWS Secret Access Key
specified, or ``aws://``).
-
Default Connection IDs
-----------------------
The default connection ID is ``aws_default``. If the environment/machine where you are running Airflow has the
file credentials in ``/home/.aws/``, and the default connection has user and pass fields empty, it will take
automatically the credentials from there.
-.. note:: Previously, the ``aws_default`` connection had the "extras" field set to ``{"region_name": "us-east-1"}``
+.. important:: Previously, the ``aws_default`` connection had the "extras" field set to ``{"region_name": "us-east-1"}``
on install. This means that by default the ``aws_default`` connection used the ``us-east-1`` region.
This is no longer the case and the region needs to be set manually, either in the connection screens in Airflow,
or via the ``AWS_DEFAULT_REGION`` environment variable.
+.. caution:: If you do not set ``[database] load_default_connections`` to ``True``
+ most probably you do not have ``aws_default``. By historical reason Amazon Provider
+ components (Hooks, Operators, Sensors, etc.) fallback to default boto3 credentials strategy
+ in case of missing Connection ID, this behaviour is deprecated and will be removed in a future releases.
Review Comment:
```suggestion
components (Hooks, Operators, Sensors, etc.) fallback to the default boto3 credentials strategy
in case of a missing Connection ID. This behaviour is deprecated and will be removed in a future releases.
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -32,19 +45,26 @@ Alternatively, one can pass credentials in as a Connection initialisation parame
To use IAM instance profile, create an "empty" connection (i.e. one with no AWS Access Key ID or AWS Secret Access Key
specified, or ``aws://``).
-
Default Connection IDs
-----------------------
The default connection ID is ``aws_default``. If the environment/machine where you are running Airflow has the
file credentials in ``/home/.aws/``, and the default connection has user and pass fields empty, it will take
automatically the credentials from there.
-.. note:: Previously, the ``aws_default`` connection had the "extras" field set to ``{"region_name": "us-east-1"}``
+.. important:: Previously, the ``aws_default`` connection had the "extras" field set to ``{"region_name": "us-east-1"}``
on install. This means that by default the ``aws_default`` connection used the ``us-east-1`` region.
This is no longer the case and the region needs to be set manually, either in the connection screens in Airflow,
or via the ``AWS_DEFAULT_REGION`` environment variable.
+.. caution:: If you do not set ``[database] load_default_connections`` to ``True``
+ most probably you do not have ``aws_default``. By historical reason Amazon Provider
+ components (Hooks, Operators, Sensors, etc.) fallback to default boto3 credentials strategy
+ in case of missing Connection ID, this behaviour is deprecated and will be removed in a future releases.
+
+ If you need use default boto3 credential strategy (credentials in environment variables, IAM Profile, etc.)
+ please provide ``None`` instead of connection id.
Review Comment:
```suggestion
If you need to use the default boto3 credential strategy (credentials in environment variables, IAM Profile, etc.)
please provide ``None`` instead of a connection ID.
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -20,8 +20,21 @@
Amazon Web Services Connection
==============================
-The Amazon Web Services connection type enables the :ref:`AWS Integrations
-<AWS>`.
+The Amazon Web Services connection type enables the :ref:`AWS Integrations <AWS>`.
+
+.. important:: Amazon Web Services Connection could be tested in the UI/API or by call
+ :meth:`~airflow.providers.amazon.aws.hooks.base_aws.AwsGenericHook.test_connection`,
+ it is **important** to correct interpret result of this test.
+ During this test components of Amazon Provider invoke AWS Security Token Service API
+ `GetCallerIdentity <https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html>`__.
+ This service **only** could check is your credentials valid or not.
+ Unfortunately it is not possible to validate is this credentials has access to specific AWS service or not.
+
+ If you use Amazon Provider for communicate with AWS API compatible services (MinIO, LocalStack, etc.)
Review Comment:
```suggestion
If you use the Amazon Provider to communicate with AWS API compatible services (MinIO, LocalStack, etc.)
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -20,8 +20,21 @@
Amazon Web Services Connection
==============================
-The Amazon Web Services connection type enables the :ref:`AWS Integrations
-<AWS>`.
+The Amazon Web Services connection type enables the :ref:`AWS Integrations <AWS>`.
+
+.. important:: Amazon Web Services Connection could be tested in the UI/API or by call
+ :meth:`~airflow.providers.amazon.aws.hooks.base_aws.AwsGenericHook.test_connection`,
+ it is **important** to correct interpret result of this test.
+ During this test components of Amazon Provider invoke AWS Security Token Service API
+ `GetCallerIdentity <https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html>`__.
+ This service **only** could check is your credentials valid or not.
+ Unfortunately it is not possible to validate is this credentials has access to specific AWS service or not.
+
+ If you use Amazon Provider for communicate with AWS API compatible services (MinIO, LocalStack, etc.)
+ test connection failure **doesn't mean** that your connection has wrong credentials.
+ Many of compatible services provide only limited amount of AWS API services,
+ and most of them not implement AWS STS GetCallerIdentity method.
Review Comment:
```suggestion
Many compatible services provide only a limited number of AWS API services,
and most of them do not implement the AWS STS GetCallerIdentity method.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29341: Add clarification about AWS Connection behaviour
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #29341:
URL: https://github.com/apache/airflow/pull/29341#issuecomment-1416266795
> Do we have a way to link this doc to the test button of Amazon provider?
I tried to find this way in last summer but better that I could find it was placeholder to description unfortunetly it is useless because it quite difficult to copy this value without DevTools.
I've also tried custom fields and put it in description, but without any luck to make this URL clickable.
I think that probably there is exists better way to solve a lot of problem with Connections, maybe we should make it pluggable and decompose it out of the Hooks, hooks still should use them but as separate object which initialise everything required behind the hook itself, and make it more extensible, self documented and provide better custom fields support rather than now. But it just my thoughts, I tried to find a more reliable way since I try to implement ECR login to docker hook.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas merged pull request #29341: Add clarification about AWS Connection behaviour
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas merged PR #29341:
URL: https://github.com/apache/airflow/pull/29341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #29341: Add clarification about AWS Connection behaviour
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #29341:
URL: https://github.com/apache/airflow/pull/29341#discussion_r1095044908
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -20,8 +20,21 @@
Amazon Web Services Connection
==============================
-The Amazon Web Services connection type enables the :ref:`AWS Integrations
-<AWS>`.
+The Amazon Web Services connection type enables the :ref:`AWS Integrations <AWS>`.
+
+.. important:: Amazon Web Services Connection could be tested in the UI/API or by call
+ :meth:`~airflow.providers.amazon.aws.hooks.base_aws.AwsGenericHook.test_connection`,
+ it is **important** to correct interpret result of this test.
+ During this test components of Amazon Provider invoke AWS Security Token Service API
+ `GetCallerIdentity <https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html>`__.
+ This service **only** could check is your credentials valid or not.
Review Comment:
```suggestion
This service can **only** check if your credentials are valid.
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -20,8 +20,21 @@
Amazon Web Services Connection
==============================
-The Amazon Web Services connection type enables the :ref:`AWS Integrations
-<AWS>`.
+The Amazon Web Services connection type enables the :ref:`AWS Integrations <AWS>`.
+
+.. important:: Amazon Web Services Connection could be tested in the UI/API or by call
+ :meth:`~airflow.providers.amazon.aws.hooks.base_aws.AwsGenericHook.test_connection`,
+ it is **important** to correct interpret result of this test.
+ During this test components of Amazon Provider invoke AWS Security Token Service API
+ `GetCallerIdentity <https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html>`__.
+ This service **only** could check is your credentials valid or not.
+ Unfortunately it is not possible to validate is this credentials has access to specific AWS service or not.
Review Comment:
```suggestion
Unfortunately it is not possible to validate if credentials have access to specific AWS service or not.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org