(airflow) branch main updated: Speed up CI image restores with zstd-compressed stashes (#73997)

[sh...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

shahar1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new 53e43f29130 Speed up CI image restores with zstd-compressed stashes (#73997)
53e43f29130 is described below

commit 53e43f291303e43018ac51a0f21236514ae62485
Author: Yossi Eliaz <zo...@users.noreply.github.com>
AuthorDate: Thu Oct 1 19:31:58 2026 +0300

    Speed up CI image restores with zstd-compressed stashes (#73997)
    
    Every CI-image consumer downloads the stash as a zlib-6 zip and inflates
    about 8 GB before docker image load can start, and the producer spends
    135-200 s zipping it. Compressing the tar once with zstd and storing it
    without zlib removes the inflate and moves about 12% fewer bytes, so
    restore+load per consumer drops by about 50 s and the producer gets
    faster too.
    
    The file keeps its name and stash key because docker image load detects
    zstd by content: consumers on any branch, including release branches
    whose Breeze only accepts ci-image-save-v3-*.tar, load it unchanged.
---
 .github/workflows/ci-image-build.yml | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/.github/workflows/ci-image-build.yml b/.github/workflows/ci-image-build.yml
index 7817fd211c3..1541337bc5c 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -321,12 +321,18 @@ jobs:
           docker rmi "${CACHE_FROM_IMAGE}"
         shell: bash
         if: always() && env.CACHE_FROM_IMAGE != ''
+      # zstd stands in for the stash's zlib (compression-level 0 below), which every consumer inflates.
+      # The .tar name, which every Breeze expects, stays: docker image load detects zstd by content.
       - name: "Export CI docker image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
         env:
           PLATFORM: ${{ inputs.platform }}
           COMMIT_FILE: "/mnt/ci-image-commit-${{ env.PYTHON_MAJOR_MINOR_VERSION }}.txt"
         run: |
           breeze ci-image save --platform "${PLATFORM}" --image-file-dir "/mnt"
+          for image_file in /mnt/ci-image-save-*-"${PYTHON_MAJOR_MINOR_VERSION}".tar; do
+            zstd -6 -T0 --rm -o "${image_file}.zst" "${image_file}"
+            mv "${image_file}.zst" "${image_file}"
+          done
           git rev-parse HEAD > "${COMMIT_FILE}"
         shell: bash
         if: inputs.upload-image-artifact == 'true' && steps.stashed-image.outputs.reusable != 'true'
@@ -336,6 +342,7 @@ jobs:
           key: ci-image-save-v3-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}
           path: "/mnt/ci-image-save-*-${{ env.PYTHON_MAJOR_MINOR_VERSION }}.tar"
           if-no-files-found: 'error'
+          compression-level: '0'
           retention-days: '2'
         if: inputs.upload-image-artifact == 'true' && inputs.image-stash-ref == ''
       # The per-branch stash above is written by every build on this branch, so an image built
@@ -350,6 +357,7 @@ jobs:
             -${{ inputs.image-stash-ref }}"
           path: "/mnt/ci-image-save-*-${{ env.PYTHON_MAJOR_MINOR_VERSION }}.tar"
           if-no-files-found: 'error'
+          compression-level: '0'
           retention-days: '6'
         if: >
           inputs.upload-image-artifact == 'true' && inputs.image-stash-ref != '' &&