(airflow) branch v2-8-test updated: Fix intermittent failures coming from buildkit parallel builds

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/v2-8-test by this push:
     new 5f078a5f67 Fix intermittent failures coming from buildkit parallel builds
5f078a5f67 is described below

commit 5f078a5f67c0e969ffdeb6db5d7181b7189e3d92
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Feb 13 22:11:23 2024 +0100

    Fix intermittent failures coming from buildkit parallel builds
    
    The PROD cachie build often fails (especially in v2-8-test) when it
    tries to rebuild the PROD cache in parallel on ARM. There is some weird
    inter-buildx problem with it and some people experience it sometimes
    as documented in https://github.com/moby/buildkit/issues/2367
    
    Instead of finding the root cause, we change this specific job in
    CI to run sequentially. This changes the time it will take to update
    cache - but not as much as 4x because the builds do not parallelise
    very well anyway. Likely instead of 8m we will get maybe total 15m,
    and since this is just cache update after everything else has
    completed, it does not matter too much if it runs a little longer.
    
    (cherry picked from commit cb63334575c1b46e54d44f9b5b4721d3f919a733)
---
 .github/workflows/ci.yml | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 9d3f5c41ac..3e941340b9 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -2519,14 +2519,14 @@ jobs:
       - name: "Copy dist packages to docker-context files"
         run: cp -v --no-preserve=mode,ownership ./dist/*.whl ./docker-context-files
       - name: "Push PROD cache ${{ matrix.python-version }} ${{ matrix.platform }}"
-        run: >
-          breeze prod-image build
-          --builder airflow_cache
-          --install-packages-from-context
-          --run-in-parallel
-          --airflow-constraints-mode constraints-source-providers
-          --prepare-buildx-cache
-          --platform ${{ matrix.platform }}
+        run: |
+          # Do not run parallel builds here as they often fail due to github token expiry issues similar to
+          # those described in https://github.com/moby/buildkit/issues/2367
+          for python in ${{needs.build-info.outputs.python-versions-list-as-string}}; do
+            breeze prod-image build --builder airflow_cache --install-packages-from-context \
+            --airflow-constraints-mode constraints-source-providers --prepare-buildx-cache \
+            --platform ${{ matrix.platform }} --python ${python}
+          done
         env:
           COMMIT_SHA: ${{ github.sha }}
       - name: "Push PROD latest image ${{ matrix.platform }}"