[airflow-ci-infra] branch make-docker-login-more-robust created (now 191c31b)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch make-docker-login-more-robust
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git.


      at 191c31b  Runners more resiliant to docker login failure

This branch includes the following new commits:

     new 191c31b  Runners more resiliant to docker login failure

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[airflow-ci-infra] 01/01: Runners more resiliant to docker login failure

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch make-docker-login-more-robust
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git

commit 191c31bf2f50642b81fe3e397f8de4ad62d3be27
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Mar 20 14:15:15 2021 +0100

    Runners more resiliant to docker login failure
    
    Login to docker registry is now done in PreExec and in case it
    fails, it also fails the whole service (leading to subsequent
    service restart).
    
    Also added `set -eux -o pipefail` to be better protected against
    any silent failures.
---
 cloud-init.yml | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/cloud-init.yml b/cloud-init.yml
index d1d0b42..68a9280 100644
--- a/cloud-init.yml
+++ b/cloud-init.yml
@@ -30,6 +30,7 @@ runcmd:
     - -c
     # https://github.com/actions/virtual-environments/blob/525f79f479cca77aef4e0a680548b65534c64a18/images/linux/scripts/installers/docker-compose.sh
     - |
+      set -exu -o pipefail
       URL=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | jq -r '.assets[].browser_download_url | select(endswith("docker-compose-Linux-x86_64"))')
       curl -L $URL -o /usr/local/bin/docker-compose
       chmod +x /usr/local/bin/docker-compose
@@ -51,6 +52,7 @@ runcmd:
     - bash
     - -c
     - |
+      set -exu -o pipefail
       python3 -mvenv /opt/runner-supervisor
       /opt/runner-supervisor/bin/pip install -U pip python-dynamodb-lock-whatnick==0.9.3 click==7.1.2 psutil 'tenacity~=6.0'
   -
@@ -76,10 +78,6 @@ runcmd:
       aws s3 cp s3://airflow-ci-assets/runner-supervisor.py /opt/runner-supervisor/bin/runner-supervisor
       chmod 755 /opt/runner-supervisor/bin/runner-supervisor
 
-      # Log in to a paid docker user to get unlimited docker pulls
-      aws ssm get-parameter --with-decryption --name /runners/apache/airflow/dockerPassword | \
-        jq .Parameter.Value -r | \
-        sudo -u runner docker login --username airflowcirunners --password-stdin
     - 2.277.1-airflow3
   - [systemctl, enable, --now, iptables.service]
   # Restart docker after applying the user firewall -- else some rules/chains might be list!
@@ -90,8 +88,8 @@ runcmd:
     - bash
     - -c
     - |
-      echo "Pre-loading commonly used docker images from S3"
       set -eux -o pipefail
+      echo "Pre-loading commonly used docker images from S3"
       aws s3 cp s3://airflow-ci-assets/pre-baked-images.tar.gz - | docker load
 
 write_files:
@@ -122,11 +120,17 @@ write_files:
   # Don't put this in ~runner, as these get written before the user is added, and this messes up creating the home dir
   - path: /usr/local/sbin/runner-cleanup-workdir.sh
     content: |
-      #!/bin/bash
+      #!/bin/bashq
+      set -exu -o pipefail
       echo "Left-over containers:"
       docker ps -a
       docker ps -qa | xargs --verbose --no-run-if-empty docker rm -fv
 
+      echo "Log in to a paid docker user to get unlimited docker pulls"
+      aws ssm get-parameter --with-decryption --name /runners/apache/airflow/dockerPassword | \
+        jq .Parameter.Value -r | \
+        sudo -u runner docker login --username airflowcirunners --password-stdin
+
       if [[ -d ~runner/actions-runner/_work/airflow/airflow ]]; then
         cd ~runner/actions-runner/_work/airflow/airflow
 
@@ -145,7 +149,7 @@ write_files:
   - path: /usr/local/bin/stop-runner-if-no-job.sh
     content: |
       #!/bin/bash
-      set -u
+      set -exu -o pipefail
 
       MAINPID="${MAINPID:-${1:-}}"
 
@@ -195,7 +199,7 @@ write_files:
     permissions: '0775'
     content: |
       #!/bin/bash
-
+      set -exu -o pipefail
       if pgrep -c Runner.Worker >/dev/null; then
           # Only report metric when we're doing something -- no point paying to submit zeros
           aws cloudwatch put-metric-data --metric-name jobs-running --value "$(pgrep -c Runner.Worker)" --namespace github.actions