[airflow] branch main updated: Fix cache builing on CI and building CI images locally (#32933)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new 1f21e44bea Fix cache builing on CI and building CI images locally (#32933)
1f21e44bea is described below

commit 1f21e44beab1923b7a8f097b8d044d4159236c5d
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Jul 29 09:56:29 2023 +0200

    Fix cache builing on CI and building CI images locally (#32933)
    
    Buildig cache on our CI has been broken since we added open
    lineage provider with >=2.7.0 limit. This should be handled by
    also adding VERSION_SUFFIX_FOR_PYPI to the cache builds (which
    is anyhow needed because otherwise cache would not be really
    functional.
    
    We actually also have to add it to ci-image-build automatically
    in this case otherwise users won't be able to build their images
    locally.
---
 .github/workflows/build-images.yml                          | 1 +
 .github/workflows/ci.yml                                    | 2 ++
 dev/breeze/src/airflow_breeze/commands/ci_image_commands.py | 9 +++++++--
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index 82a857bdab..daa94ab290 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -214,6 +214,7 @@ jobs:
       DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
       RUNS_ON: ${{ needs.build-info.outputs.runs-on }}
       BACKEND: sqlite
+      VERSION_SUFFIX_FOR_PYPI: "dev0"
     steps:
       - name: Cleanup repo
         run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index b152bff759..8c8cf44015 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -207,6 +207,7 @@ jobs:
     env:
       RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
       UPGRADE_TO_NEWER_DEPENDENCIES: false
+      VERSION_SUFFIX_FOR_PYPI: "dev0"
     continue-on-error: true
     steps:
       - name: Cleanup repo
@@ -1628,6 +1629,7 @@ jobs:
         platform: ["linux/amd64", "linux/arm64"]
     env:
       RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
+      VERSION_SUFFIX_FOR_PYPI: "dev0"
     steps:
       - name: Cleanup repo
         run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
index 141908864e..9dacbc0953 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
@@ -460,11 +460,16 @@ def run_build_ci_image(
       * update cached information that the build completed and saves checksums of all files
         for quick future check if the build is needed
 
-
-
     :param ci_image_params: CI image parameters
     :param output: output redirection
     """
+    if not ci_image_params.version_suffix_for_pypi:
+        # We need that to handle the >= 2.7.0 limit we have for openlineage provider at least until
+        # Airflow 2.7.0 release is out, in order to avoid conflicting dependencies while building the image
+        # We are setting version_suffix_for_pypi to dev0 for CI builds where cache is prepared, so in
+        # order to have the cache used effectively, we should also locally force the version_suffix_for_pypi
+        # to dev0. We might evan leave it as default value in the future (to be decided after 2.7.0 release)
+        ci_image_params.version_suffix_for_pypi = "dev0"
     if (
         ci_image_params.is_multi_platform()
         and not ci_image_params.push