[GitHub] [airflow] potiuk commented on a change in pull request #10146: Retry max 3 times if failing to initialize integration

[GitBox <gi...@apache.org>]

potiuk commented on a change in pull request #10146:
URL: https://github.com/apache/airflow/pull/10146#discussion_r464871921



##########
File path: scripts/ci/testing/ci_run_airflow_testing.sh
##########
@@ -31,14 +31,22 @@ fi
 function run_airflow_testing_in_docker() {
     set +u
     set +e
-    # shellcheck disable=SC2016
-    docker-compose --log-level INFO \
-      -f "${SCRIPTS_CI_DIR}/docker-compose/base.yml" \
-      -f "${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml" \
-      "${INTEGRATIONS[@]}" \
-      "${DOCKER_COMPOSE_LOCAL[@]}" \
-         run airflow "${@}"
-    EXIT_CODE=$?
+    for TRY_NUM in {1..3}
+    do
+        docker-compose --log-level INFO \
+          -f "${SCRIPTS_CI_DIR}/docker-compose/base.yml" \
+          -f "${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml" \
+          "${INTEGRATIONS[@]}" \
+          "${DOCKER_COMPOSE_LOCAL[@]}" \
+             run airflow "${@}"
+        EXIT_CODE=$?
+        if [[ ${EXIT_CODE} == 254 ]]; then
+            echo "Failed on starting integration on ${TRY_NUM} try."

Review comment:
       @feluelle Yeah. It won't change too much but sleeping a few seconds will help with flushing any buffers etc. 
   Also I added one more step - I run 'docker-compose down --remove-orphans -v' which will shut down the network and anything else that remained from the previous docker-compose run (but the images remain). This way if an integration started already - it will be killed.
   
   I think one part of the problem is an occasional race that happens between establishing the network by docker-compose and starting the integrations - when the images are being downloaded. So the second (and third) run of docker-compose will start with all images already downloaded and completely new network to set-up.
   
   I think that should help.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org