(airflow) 41/44: Improve speed of regular upgrades when dependencies change. (#37360)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 5b0e621d71f01ad55fb598f270be9eb971cae99d
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Feb 12 21:30:54 2024 +0100

    Improve speed of regular upgrades when dependencies change. (#37360)
    
    When regular dependencies change we enable UPGRADE_TO_NEWER_DEPENDENCIES
    flag. After some recent improvements in installing dependencies from
    the branch tip where we can install airflow using `[devel-ci]` extra
    from the branch tip, this opens up another option for speeding up
    the installations, when some dependencies change.
    
    So far - UPGRADE_TO_NEWER_DEPENDENCIES necessitated to reinstall all
    dependencies with airflow attempting to upgrade them eagerly. This
    takes ~ 28 minutes on CI now from the scratch. However, this is not
    needed. We can still start from the set of dependencies pre-installed
    by installation from branch tip, and only trigger
    "UPGRADE_TO_NEWER_DEPENDENCIES" after that - so that the "URL"
    installation cache does not get invalidated.
    
    This should limit the time needed to build such an image to ~ 8 minutes
    in CI when cache is built.
    
    In the future we might further optimize it - as we now have all the
    mechanisms necessary to do that in selective checks. For example we
    could trigger UPGRADE_TO_NEWER_DEPENDENCIES only when some dependencies
    are changes, and not when new dependencies are added, further limiting
    the time needed to build such image to some ~ 3/4 minutes. But this
    should be a separate PR and should be carefuly thought about, as
    it might also be that this will lead to conflicts with the latest
    constraints..
    
    (cherry picked from commit d8a42cadf85df026f4e41e5e440e350380615f4a)
---
 Dockerfile.ci | 37 ++++++++++++++++++-------------------
 1 file changed, 18 insertions(+), 19 deletions(-)

diff --git a/Dockerfile.ci b/Dockerfile.ci
index 6227af0640..d818dc18e8 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -1126,23 +1126,6 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
 
 RUN echo "Airflow version: ${AIRFLOW_VERSION}"
 
-# Those are additional constraints that are needed for some extras but we do not want to
-# force them on the main Airflow package. Currently we need no extra limits as PIP 23.1+ has much better
-# dependency resolution and we do not need to limit the versions of the dependencies
-#
-# boto3 is limited to <1.34 because of aiobotocore that only works with 1.33 and we want to help
-# `pip` to limit the versions it checks and limit backtracking, by explicitly specifying these limits
-# when performing eager upgrade of dependencies - this way it won't even consider 1.34 versions of boto
-# We should update it every time a new version of aiobotocore is released supporting 1.34
-#
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="boto3>=1.33,<1.34"
-ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
-ARG VERSION_SUFFIX_FOR_PYPI=""
-
-ENV EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS} \
-    UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES} \
-    VERSION_SUFFIX_FOR_PYPI=${VERSION_SUFFIX_FOR_PYPI}
-
 # Copy all scripts required for installation - changing any of those should lead to
 # rebuilding from here
 COPY --from=scripts install_pip_version.sh install_airflow_dependencies_from_branch_tip.sh \
@@ -1157,8 +1140,7 @@ COPY --from=scripts install_pip_version.sh install_airflow_dependencies_from_bra
 # the cache is only used when "upgrade to newer dependencies" is not set to automatically
 # account for removed dependencies (we do not install them in the first place)
 RUN bash /scripts/docker/install_pip_version.sh; \
-    if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" && \
-          ${UPGRADE_TO_NEWER_DEPENDENCIES} == "false" ]]; then \
+    if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" ]]; then \
         bash /scripts/docker/install_airflow_dependencies_from_branch_tip.sh; \
     fi
 
@@ -1183,6 +1165,23 @@ COPY airflow_pre_installed_providers.txt ${AIRFLOW_SOURCES}/
 COPY hatch_build.py ${AIRFLOW_SOURCES}/
 COPY --from=scripts install_airflow.sh /scripts/docker/
 
+# Those are additional constraints that are needed for some extras but we do not want to
+# force them on the main Airflow package. Currently we need no extra limits as PIP 23.1+ has much better
+# dependency resolution and we do not need to limit the versions of the dependencies
+#
+# boto3 is limited to <1.34 because of aiobotocore that only works with 1.33 and we want to help
+# `pip` to limit the versions it checks and limit backtracking, by explicitly specifying these limits
+# when performing eager upgrade of dependencies - this way it won't even consider 1.34 versions of boto
+# We should update it every time a new version of aiobotocore is released supporting 1.34
+#
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="boto3>=1.33,<1.34"
+ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
+ARG VERSION_SUFFIX_FOR_PYPI=""
+
+ENV EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS} \
+    UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES} \
+    VERSION_SUFFIX_FOR_PYPI=${VERSION_SUFFIX_FOR_PYPI}
+
 # The goal of this line is to install the dependencies from the most current pyproject.toml from sources
 # This will be usually incremental small set of packages in CI optimized build, so it will be very fast
 # In non-CI optimized build this will install all dependencies before installing sources.