[GitHub] [airflow] potiuk commented on a change in pull request #15438: Speeds up Docker build process by combining RUN and ENV commands

[GitBox <gi...@apache.org>]

potiuk commented on a change in pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#discussion_r616502878



##########
File path: Dockerfile
##########
@@ -139,98 +133,94 @@ RUN mkdir -pv /usr/share/man/man1 \
     && rm -rf /var/lib/apt/lists/*
 
 ARG INSTALL_MYSQL_CLIENT="true"
-ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT}
-
-# Only copy install_mysql.sh to not invalidate cache on other script changes
-COPY scripts/docker/install_mysql.sh /scripts/docker/install_mysql.sh
-COPY docker-context-files /docker-context-files
-# fix permission issue in Azure DevOps when running the script
-RUN bash ./scripts/docker/install_mysql.sh dev
-
 ARG AIRFLOW_REPO=apache/airflow
-ENV AIRFLOW_REPO=${AIRFLOW_REPO}
-
 ARG AIRFLOW_BRANCH=master
-ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH}
-
 ARG AIRFLOW_EXTRAS
 ARG ADDITIONAL_AIRFLOW_EXTRAS=""
-ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
-
 # Allows to override constraints source
 ARG CONSTRAINTS_GITHUB_REPOSITORY="apache/airflow"
-ENV CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY}
-
 ARG AIRFLOW_CONSTRAINTS="constraints"
-ENV AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS}
 ARG AIRFLOW_CONSTRAINTS_REFERENCE=""
-ENV AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE}
 ARG AIRFLOW_CONSTRAINTS_LOCATION=""
-ENV AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}
-
 ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-master"
-ENV DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}
-
-ENV PATH=${PATH}:/root/.local/bin
-RUN mkdir -p /root/.local/bin
-
-RUN if [[ -f /docker-context-files/.pypirc ]]; then \
-        cp /docker-context-files/.pypirc /root/.pypirc; \
-    fi
-
 ARG AIRFLOW_PIP_VERSION
-ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
-
 # By default PIP has progress bar but you can disable it.
 ARG PIP_PROGRESS_BAR
-ENV PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR}
-
-# Install Airflow with "--user" flag, so that we can copy the whole .local folder to the final image
-# from the build image and always in non-editable mode
-ENV AIRFLOW_INSTALL_USER_FLAG="--user"
-ENV AIRFLOW_INSTALL_EDITABLE_FLAG=""
-
-# Upgrade to specific PIP version
-RUN pip install --no-cache-dir --upgrade "pip==${AIRFLOW_PIP_VERSION}"
-
 # By default we do not use pre-cached packages, but in CI/Breeze environment we override this to speed up
 # builds in case setup.py/setup.cfg changed. This is pure optimisation of CI/Breeze builds.
 ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
-ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES}
-
+# This is airflow version that is put in the label of the image build
+ARG AIRFLOW_VERSION
 # By default we install providers from PyPI but in case of Breeze build we want to install providers
 # from local sources without the need of preparing provider packages upfront. This value is
 # automatically overridden by Breeze scripts.
 ARG INSTALL_PROVIDERS_FROM_SOURCES="false"
-ENV INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES}
-
-# This is airflow version that is put in the label of the image build
-ARG AIRFLOW_VERSION
-ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
-
 # Determines the way airflow is installed. By default we install airflow from PyPI `apache-airflow` package
 # But it also can be `.` from local installation or GitHub URL pointing to specific branch or tag
 # Of Airflow. Note That for local source installation you need to have local sources of
 # Airflow checked out together with the Dockerfile and AIRFLOW_SOURCES_FROM and AIRFLOW_SOURCES_TO
 # set to "." and "/opt/airflow" respectively.
 ARG AIRFLOW_INSTALLATION_METHOD="apache-airflow"
-ENV AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD}
-
 # By default latest released version of airflow is installed (when empty) but this value can be overridden
 # and we can install version according to specification (For example ==2.0.2 or <3.0.0).
 ARG AIRFLOW_VERSION_SPECIFICATION=""
-ENV AIRFLOW_VERSION_SPECIFICATION=${AIRFLOW_VERSION_SPECIFICATION}
+# By default we do not upgrade to latest dependencies
+ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
+# By default we install latest airflow from PyPI so we do not need to copy sources of Airflow
+# but in case of breeze/CI builds we use latest sources and we override those
+# those SOURCES_FROM/TO with "." and "/opt/airflow" respectively
+ARG AIRFLOW_SOURCES_FROM="empty"
+ARG AIRFLOW_SOURCES_TO="/empty"
+
+
+ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \
+    AIRFLOW_REPO=${AIRFLOW_REPO} \
+    AIRFLOW_BRANCH=${AIRFLOW_BRANCH} \
+    AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS} \
+    CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY} \
+    AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS} \
+    AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \
+    AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \
+    DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \
+    PATH=${PATH}:/root/.local/bin \
+    AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
+    PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR} \
+    # Install Airflow with "--user" flag, so that we can copy the whole .local folder to the final image
+    # from the build image and always in non-editable mode
+    AIRFLOW_INSTALL_USER_FLAG="--user" \
+    AIRFLOW_INSTALL_EDITABLE_FLAG="" \
+    UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES}
+
+# Only copy install_mysql.sh to not invalidate cache on other script changes
+COPY scripts/docker/install_mysql.sh /scripts/docker/install_mysql.sh
+COPY docker-context-files /docker-context-files

Review comment:
       It can be moved few lines down. Good catch.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org