[airflow] 14/34: Better diagnostics for image waiting (#14779)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v2-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 29f098eedbe82079124f2de8bf65d2ff4cb2c621
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Mar 14 20:16:04 2021 +0100

    Better diagnostics for image waiting (#14779)
    
    Sometimes (very rarely) some 'wait for image' pulling steps
    loop forever (while other steps from parallell jobs pulling the
    same image have no problems).
    
    Example here:
    
    Failed step here:
    
    * https://github.com/apache/airflow/runs/2106723280?check_suite_focus=true#step:5:349
    
    Another similar step in parallel job had no problems with retrieving the
    same image earlier:
    
    * https://github.com/apache/airflow/runs/2106723269?check_suite_focus=true#step:5:119
    
    Both images pulled the same image:
    
    docker.pkg.github.com/apache/airflow/master-python3.6-ci-v2:651461418
    
    This change adds diagnostics information that might provide more
    information in case this happens again so that we can understand
    what is going on and mitigate the issue.
    
    (cherry picked from commit 4cde47b339d5ab5e4ebccfb22962ccbfc4c5457e)
---
 scripts/ci/libraries/_build_images.sh | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index be4833d..2deb574 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -998,11 +998,18 @@ function build_images::wait_for_image_tag() {
     start_end::group_start "Wait for image tag ${IMAGE_TO_WAIT_FOR}"
     while true; do
         set +e
-        docker pull "${IMAGE_TO_WAIT_FOR}" 2>/dev/null >/dev/null
+        echo "${COLOR_BLUE}Docker pull ${IMAGE_TO_WAIT_FOR} ${COLOR_RESET}" >"${OUTPUT_LOG}"
+        docker pull "${IMAGE_TO_WAIT_FOR}" >>"${OUTPUT_LOG}" 2>&1
         set -e
-        if [[ -z "$(docker images -q "${IMAGE_TO_WAIT_FOR}" 2>/dev/null || true)" ]]; then
+        local image_hash
+        echo "${COLOR_BLUE} Docker images -q ${IMAGE_TO_WAIT_FOR}${COLOR_RESET}" >>"${OUTPUT_LOG}"
+        image_hash="$(docker images -q "${IMAGE_TO_WAIT_FOR}" 2>>"${OUTPUT_LOG}" || true)"
+        if [[ -z "${image_hash}" ]]; then
             echo
-            echo "The image ${IMAGE_TO_WAIT_FOR} is not yet available. Waiting"
+            echo "The image ${IMAGE_TO_WAIT_FOR} is not yet available. No local hash for the image. Waiting."
+            echo
+            echo "Last log:"
+            cat "${OUTPUT_LOG}"
             echo
             sleep 10
         else