You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/24 23:23:44 UTC
[GitHub] [airflow] Taragolis opened a new pull request, #23906: docs: amazon-provider retry modes
Taragolis opened a new pull request, #23906:
URL: https://github.com/apache/airflow/pull/23906
Added information how to reduce chance of boto3 throttling exceptions in amazon-provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #23906: docs: amazon-provider retry modes
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23906:
URL: https://github.com/apache/airflow/pull/23906#issuecomment-1147418922
NICE!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #23906: docs: amazon-provider retry modes
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23906:
URL: https://github.com/apache/airflow/pull/23906#discussion_r881227483
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -194,6 +194,69 @@ The following settings may be used within the ``assume_role_with_saml`` containe
https://pypi.org/project/requests-gssapi/
+Avoid Throttling exceptions
+---------------------------
+
+Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
+``apache-airflow-providers-amazon`` components might fail during execution with one of
+Throttling exception e.g.: *ThrottlingException*, *ProvisionedThroughputExceededException*, etc.
Review Comment:
```suggestion
Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
``apache-airflow-providers-amazon`` components might fail during execution with a
throttling exception, e.g. *ThrottlingException*, *ProvisionedThroughputExceededException*.
```
Grammar police! 👮
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #23906: docs: amazon-provider retry modes
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23906:
URL: https://github.com/apache/airflow/pull/23906#discussion_r881228222
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -194,6 +194,69 @@ The following settings may be used within the ``assume_role_with_saml`` containe
https://pypi.org/project/requests-gssapi/
+Avoid Throttling exceptions
+---------------------------
+
+Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
+``apache-airflow-providers-amazon`` components might fail during execution with one of
+Throttling exception e.g.: *ThrottlingException*, *ProvisionedThroughputExceededException*, etc.
+
+``botocore.config.Config`` supports different exponential backoff modes out of the box:
+``legacy``, ``standard``, ``adaptive``
+
+By default ``botocore.config.Config`` uses ``legacy`` mode with ``5`` maximum retry attempts, which in some cases not enough.
+
+If you have been faced with Throttling exceptions you've might as well
+change mode to ``standard`` with more retry attempts
Review Comment:
```suggestion
If you have been faced with throttling exceptions, you may
change the mode to ``standard`` with more retry attempts.
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -194,6 +194,69 @@ The following settings may be used within the ``assume_role_with_saml`` containe
https://pypi.org/project/requests-gssapi/
+Avoid Throttling exceptions
+---------------------------
+
+Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
+``apache-airflow-providers-amazon`` components might fail during execution with one of
+Throttling exception e.g.: *ThrottlingException*, *ProvisionedThroughputExceededException*, etc.
+
+``botocore.config.Config`` supports different exponential backoff modes out of the box:
+``legacy``, ``standard``, ``adaptive``
+
+By default ``botocore.config.Config`` uses ``legacy`` mode with ``5`` maximum retry attempts, which in some cases not enough.
+
+If you have been faced with Throttling exceptions you've might as well
+change mode to ``standard`` with more retry attempts
Review Comment:
```suggestion
If you encounter throttling exceptions, you may
change the mode to ``standard`` with more retry attempts.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #23906: docs: amazon-provider retry modes
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23906:
URL: https://github.com/apache/airflow/pull/23906#issuecomment-1137436941
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #23906: docs: amazon-provider retry modes
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #23906:
URL: https://github.com/apache/airflow/pull/23906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #23906: docs: amazon-provider retry modes
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23906:
URL: https://github.com/apache/airflow/pull/23906#discussion_r881227483
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -194,6 +194,69 @@ The following settings may be used within the ``assume_role_with_saml`` containe
https://pypi.org/project/requests-gssapi/
+Avoid Throttling exceptions
+---------------------------
+
+Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
+``apache-airflow-providers-amazon`` components might fail during execution with one of
+Throttling exception e.g.: *ThrottlingException*, *ProvisionedThroughputExceededException*, etc.
Review Comment:
```suggestion
Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
``apache-airflow-providers-amazon`` components might fail during execution with one of
throttling exception e.g.: *ThrottlingException*, *ProvisionedThroughputExceededException*.
```
Grammar police! 👮
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -194,6 +194,69 @@ The following settings may be used within the ``assume_role_with_saml`` containe
https://pypi.org/project/requests-gssapi/
+Avoid Throttling exceptions
+---------------------------
+
+Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
+``apache-airflow-providers-amazon`` components might fail during execution with one of
+Throttling exception e.g.: *ThrottlingException*, *ProvisionedThroughputExceededException*, etc.
Review Comment:
```suggestion
Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
``apache-airflow-providers-amazon`` components might fail during execution with one of
throttling exception, e.g. *ThrottlingException*, *ProvisionedThroughputExceededException*.
```
Grammar police! 👮
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #23906: docs: amazon-provider retry modes
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23906:
URL: https://github.com/apache/airflow/pull/23906#discussion_r881227943
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -194,6 +194,69 @@ The following settings may be used within the ``assume_role_with_saml`` containe
https://pypi.org/project/requests-gssapi/
+Avoid Throttling exceptions
+---------------------------
+
+Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
+``apache-airflow-providers-amazon`` components might fail during execution with one of
+Throttling exception e.g.: *ThrottlingException*, *ProvisionedThroughputExceededException*, etc.
+
+``botocore.config.Config`` supports different exponential backoff modes out of the box:
+``legacy``, ``standard``, ``adaptive``
+
+By default ``botocore.config.Config`` uses ``legacy`` mode with ``5`` maximum retry attempts, which in some cases not enough.
Review Comment:
```suggestion
By default, ``botocore.config.Config`` uses ``legacy`` mode with 5 maximum retry attempts, which may not be enough in some cases.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #23906: docs: amazon-provider retry modes
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23906:
URL: https://github.com/apache/airflow/pull/23906#discussion_r881228776
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -194,6 +194,69 @@ The following settings may be used within the ``assume_role_with_saml`` containe
https://pypi.org/project/requests-gssapi/
+Avoid Throttling exceptions
+---------------------------
+
+Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
+``apache-airflow-providers-amazon`` components might fail during execution with one of
+Throttling exception e.g.: *ThrottlingException*, *ProvisionedThroughputExceededException*, etc.
+
+``botocore.config.Config`` supports different exponential backoff modes out of the box:
+``legacy``, ``standard``, ``adaptive``
+
+By default ``botocore.config.Config`` uses ``legacy`` mode with ``5`` maximum retry attempts, which in some cases not enough.
+
+If you have been faced with Throttling exceptions you've might as well
+change mode to ``standard`` with more retry attempts
+
+
+.. seealso::
+ https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#retries
+
+Set in Connection
+^^^^^^^^^^^^^^^^^
+
+**Connection extra field**:
+ .. code-block:: json
+
+ {
+ "config_kwargs": {
+ "retries": {
+ "mode": "standard",
+ "max_attempts": 10
+ }
+ }
+ }
+
+Set in AWS Config File
+^^^^^^^^^^^^^^^^^^^^^^
+
+**~/.aws/config**:
+ .. code-block:: ini
+
+ [awesome_aws_profile]
+ retry_mode = standard
+ max_attempts = 10
+
+**Connection extra field**:
+ .. code-block:: json
+
+ {
+ "session_kwargs": {
+ "profile_name": "awesome_aws_profile"
+ }
+ }
+
+Set by Environment Variables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ .. note:: This set retry mode to all connections unless another retry config explicit set to specific connection
Review Comment:
```suggestion
.. note:: This sets the retry mode of all connections, unless another retry config is explicitly set on a specific connection.
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -194,6 +194,69 @@ The following settings may be used within the ``assume_role_with_saml`` containe
https://pypi.org/project/requests-gssapi/
+Avoid Throttling exceptions
+---------------------------
+
+Amazon Web Services have quota limits for simultaneous API call as result with frequent calls
+``apache-airflow-providers-amazon`` components might fail during execution with one of
+Throttling exception e.g.: *ThrottlingException*, *ProvisionedThroughputExceededException*, etc.
+
+``botocore.config.Config`` supports different exponential backoff modes out of the box:
+``legacy``, ``standard``, ``adaptive``
+
+By default ``botocore.config.Config`` uses ``legacy`` mode with ``5`` maximum retry attempts, which in some cases not enough.
+
+If you have been faced with Throttling exceptions you've might as well
+change mode to ``standard`` with more retry attempts
+
+
+.. seealso::
+ https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#retries
+
+Set in Connection
+^^^^^^^^^^^^^^^^^
+
+**Connection extra field**:
+ .. code-block:: json
+
+ {
+ "config_kwargs": {
+ "retries": {
+ "mode": "standard",
+ "max_attempts": 10
+ }
+ }
+ }
+
+Set in AWS Config File
+^^^^^^^^^^^^^^^^^^^^^^
+
+**~/.aws/config**:
+ .. code-block:: ini
+
+ [awesome_aws_profile]
+ retry_mode = standard
+ max_attempts = 10
+
+**Connection extra field**:
+ .. code-block:: json
+
+ {
+ "session_kwargs": {
+ "profile_name": "awesome_aws_profile"
+ }
+ }
+
+Set by Environment Variables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ .. note:: This set retry mode to all connections unless another retry config explicit set to specific connection
Review Comment:
```suggestion
.. note:: This sets the retry mode on all connections, unless another retry config is explicitly set on a specific connection.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org