You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/04 00:26:15 UTC
[GitHub] [airflow] potiuk opened a new pull request #10146: Retry max 3 times if failing to initialize integration
potiuk opened a new pull request #10146:
URL: https://github.com/apache/airflow/pull/10146
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #10146: Retry max 3 times if failing to initialize integration
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #10146:
URL: https://github.com/apache/airflow/pull/10146#discussion_r464880342
##########
File path: scripts/ci/testing/ci_run_airflow_testing.sh
##########
@@ -31,14 +31,22 @@ fi
function run_airflow_testing_in_docker() {
set +u
set +e
- # shellcheck disable=SC2016
- docker-compose --log-level INFO \
- -f "${SCRIPTS_CI_DIR}/docker-compose/base.yml" \
- -f "${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml" \
- "${INTEGRATIONS[@]}" \
- "${DOCKER_COMPOSE_LOCAL[@]}" \
- run airflow "${@}"
- EXIT_CODE=$?
+ for TRY_NUM in {1..3}
+ do
+ docker-compose --log-level INFO \
+ -f "${SCRIPTS_CI_DIR}/docker-compose/base.yml" \
+ -f "${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml" \
+ "${INTEGRATIONS[@]}" \
+ "${DOCKER_COMPOSE_LOCAL[@]}" \
+ run airflow "${@}"
+ EXIT_CODE=$?
+ if [[ ${EXIT_CODE} == 254 ]]; then
+ echo "Failed on starting integration on ${TRY_NUM} try."
Review comment:
Sounds great! Let's give it a try (hopefully only 1/1) :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10146: Retry max 3 times if failing to initialize integration
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10146:
URL: https://github.com/apache/airflow/pull/10146#discussion_r464871921
##########
File path: scripts/ci/testing/ci_run_airflow_testing.sh
##########
@@ -31,14 +31,22 @@ fi
function run_airflow_testing_in_docker() {
set +u
set +e
- # shellcheck disable=SC2016
- docker-compose --log-level INFO \
- -f "${SCRIPTS_CI_DIR}/docker-compose/base.yml" \
- -f "${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml" \
- "${INTEGRATIONS[@]}" \
- "${DOCKER_COMPOSE_LOCAL[@]}" \
- run airflow "${@}"
- EXIT_CODE=$?
+ for TRY_NUM in {1..3}
+ do
+ docker-compose --log-level INFO \
+ -f "${SCRIPTS_CI_DIR}/docker-compose/base.yml" \
+ -f "${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml" \
+ "${INTEGRATIONS[@]}" \
+ "${DOCKER_COMPOSE_LOCAL[@]}" \
+ run airflow "${@}"
+ EXIT_CODE=$?
+ if [[ ${EXIT_CODE} == 254 ]]; then
+ echo "Failed on starting integration on ${TRY_NUM} try."
Review comment:
@feluelle Yeah. It won't change too much but sleeping a few seconds will help with flushing any buffers etc.
Also I added one more step - I run 'docker-compose down --remove-orphans -v' which will shut down the network and anything else that remained from the previous docker-compose run (but the images remain). This way if an integration started already - it will be killed.
I think one part of the problem is an occasional race that happens between establishing the network by docker-compose and starting the integrations - when the images are being downloaded. So the second (and third) run of docker-compose will start with all images already downloaded and completely new network to set-up.
I think that should help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #10146: Retry max 3 times if failing to initialize integration
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #10146:
URL: https://github.com/apache/airflow/pull/10146
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #10146: Retry max 3 times if failing to initialize integration
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #10146:
URL: https://github.com/apache/airflow/pull/10146#discussion_r464830382
##########
File path: scripts/ci/testing/ci_run_airflow_testing.sh
##########
@@ -31,14 +31,22 @@ fi
function run_airflow_testing_in_docker() {
set +u
set +e
- # shellcheck disable=SC2016
- docker-compose --log-level INFO \
- -f "${SCRIPTS_CI_DIR}/docker-compose/base.yml" \
- -f "${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml" \
- "${INTEGRATIONS[@]}" \
- "${DOCKER_COMPOSE_LOCAL[@]}" \
- run airflow "${@}"
- EXIT_CODE=$?
+ for TRY_NUM in {1..3}
+ do
+ docker-compose --log-level INFO \
+ -f "${SCRIPTS_CI_DIR}/docker-compose/base.yml" \
+ -f "${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml" \
+ "${INTEGRATIONS[@]}" \
+ "${DOCKER_COMPOSE_LOCAL[@]}" \
+ run airflow "${@}"
+ EXIT_CODE=$?
+ if [[ ${EXIT_CODE} == 254 ]]; then
+ echo "Failed on starting integration on ${TRY_NUM} try."
Review comment:
Should we wait some time before retrying?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10146: Retry max 3 times if failing to initialize integration
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10146:
URL: https://github.com/apache/airflow/pull/10146#issuecomment-668304914
It should help to fight with occasional failures during initialization of integrations in docker-compose
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org