You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/19 18:26:32 UTC
[GitHub] [airflow] potiuk opened a new pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
potiuk opened a new pull request #15438:
URL: https://github.com/apache/airflow/pull/15438
The Dockerfile is more "packed" and certain ARG/ENVs are in separate
parts of it but we save minutes in certain scenarios when the images
are built (especially when they are built in parallell, the
difference might be significant)
This change also removes some of the old, already unused CASS_DRIVER
ARGS and ENVS. They are not needed any more as cassandra drivers do
not require CPYTHON compilation any more.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#issuecomment-823164186
[The Workflow run](https://github.com/apache/airflow/actions/runs/766781774) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#discussion_r619212935
##########
File path: Dockerfile
##########
@@ -42,8 +42,6 @@ ARG AIRFLOW_HOME=/opt/airflow
ARG AIRFLOW_UID="50000"
ARG AIRFLOW_GID="50000"
-ARG CASS_DRIVER_BUILD_CONCURRENCY="8"
Review comment:
I restored it only for CI image as it only makes sense there. For Prod build we still want to build cassandra driver with cython if not already available, and I thin the build_concurrency, I seriously doubt someone will care too much about the potential few minutes improvements there. the variable is undocumented anyway and I am not sure it is valid still in the latest cassandra driver.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #15438:
URL: https://github.com/apache/airflow/pull/15438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#issuecomment-823162849
[The Workflow run](https://github.com/apache/airflow/actions/runs/766716817) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#discussion_r616264905
##########
File path: Dockerfile
##########
@@ -106,21 +102,19 @@ ARG DEV_APT_DEPS="\
unixodbc \
unixodbc-dev \
yarn"
-ENV DEV_APT_DEPS=${DEV_APT_DEPS}
-
ARG ADDITIONAL_DEV_APT_DEPS=""
-ENV ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS}
-
ARG DEV_APT_COMMAND="\
curl --fail --location https://deb.nodesource.com/setup_10.x | bash - \
&& curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - > /dev/null \
&& echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list"
-ENV DEV_APT_COMMAND=${DEV_APT_COMMAND}
-
ARG ADDITIONAL_DEV_APT_COMMAND="echo"
-ENV ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-ARG ADDITIONAL_DEV_APT_ENV=""
+ENV DEV_APT_DEPS=${DEV_APT_DEPS}
Review comment:
Why 2 layers? Should we combine 112 and 114 into one step?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#discussion_r616266182
##########
File path: Dockerfile
##########
@@ -139,98 +133,94 @@ RUN mkdir -pv /usr/share/man/man1 \
&& rm -rf /var/lib/apt/lists/*
ARG INSTALL_MYSQL_CLIENT="true"
-ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT}
-
-# Only copy install_mysql.sh to not invalidate cache on other script changes
-COPY scripts/docker/install_mysql.sh /scripts/docker/install_mysql.sh
-COPY docker-context-files /docker-context-files
-# fix permission issue in Azure DevOps when running the script
-RUN bash ./scripts/docker/install_mysql.sh dev
-
ARG AIRFLOW_REPO=apache/airflow
-ENV AIRFLOW_REPO=${AIRFLOW_REPO}
-
ARG AIRFLOW_BRANCH=master
-ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH}
-
ARG AIRFLOW_EXTRAS
ARG ADDITIONAL_AIRFLOW_EXTRAS=""
-ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
-
# Allows to override constraints source
ARG CONSTRAINTS_GITHUB_REPOSITORY="apache/airflow"
-ENV CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY}
-
ARG AIRFLOW_CONSTRAINTS="constraints"
-ENV AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS}
ARG AIRFLOW_CONSTRAINTS_REFERENCE=""
-ENV AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE}
ARG AIRFLOW_CONSTRAINTS_LOCATION=""
-ENV AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}
-
ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-master"
-ENV DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}
-
-ENV PATH=${PATH}:/root/.local/bin
-RUN mkdir -p /root/.local/bin
-
-RUN if [[ -f /docker-context-files/.pypirc ]]; then \
- cp /docker-context-files/.pypirc /root/.pypirc; \
- fi
-
ARG AIRFLOW_PIP_VERSION
-ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
-
# By default PIP has progress bar but you can disable it.
ARG PIP_PROGRESS_BAR
-ENV PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR}
-
-# Install Airflow with "--user" flag, so that we can copy the whole .local folder to the final image
-# from the build image and always in non-editable mode
-ENV AIRFLOW_INSTALL_USER_FLAG="--user"
-ENV AIRFLOW_INSTALL_EDITABLE_FLAG=""
-
-# Upgrade to specific PIP version
-RUN pip install --no-cache-dir --upgrade "pip==${AIRFLOW_PIP_VERSION}"
-
# By default we do not use pre-cached packages, but in CI/Breeze environment we override this to speed up
# builds in case setup.py/setup.cfg changed. This is pure optimisation of CI/Breeze builds.
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
-ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES}
-
+# This is airflow version that is put in the label of the image build
+ARG AIRFLOW_VERSION
# By default we install providers from PyPI but in case of Breeze build we want to install providers
# from local sources without the need of preparing provider packages upfront. This value is
# automatically overridden by Breeze scripts.
ARG INSTALL_PROVIDERS_FROM_SOURCES="false"
-ENV INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES}
-
-# This is airflow version that is put in the label of the image build
-ARG AIRFLOW_VERSION
-ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
-
# Determines the way airflow is installed. By default we install airflow from PyPI `apache-airflow` package
# But it also can be `.` from local installation or GitHub URL pointing to specific branch or tag
# Of Airflow. Note That for local source installation you need to have local sources of
# Airflow checked out together with the Dockerfile and AIRFLOW_SOURCES_FROM and AIRFLOW_SOURCES_TO
# set to "." and "/opt/airflow" respectively.
ARG AIRFLOW_INSTALLATION_METHOD="apache-airflow"
-ENV AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD}
-
# By default latest released version of airflow is installed (when empty) but this value can be overridden
# and we can install version according to specification (For example ==2.0.2 or <3.0.0).
ARG AIRFLOW_VERSION_SPECIFICATION=""
-ENV AIRFLOW_VERSION_SPECIFICATION=${AIRFLOW_VERSION_SPECIFICATION}
+# By default we do not upgrade to latest dependencies
+ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
+# By default we install latest airflow from PyPI so we do not need to copy sources of Airflow
+# but in case of breeze/CI builds we use latest sources and we override those
+# those SOURCES_FROM/TO with "." and "/opt/airflow" respectively
+ARG AIRFLOW_SOURCES_FROM="empty"
+ARG AIRFLOW_SOURCES_TO="/empty"
+
+
+ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \
+ AIRFLOW_REPO=${AIRFLOW_REPO} \
+ AIRFLOW_BRANCH=${AIRFLOW_BRANCH} \
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS} \
+ CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY} \
+ AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS} \
+ AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \
+ AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \
+ DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \
+ PATH=${PATH}:/root/.local/bin \
+ AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
+ PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR} \
+ # Install Airflow with "--user" flag, so that we can copy the whole .local folder to the final image
+ # from the build image and always in non-editable mode
+ AIRFLOW_INSTALL_USER_FLAG="--user" \
+ AIRFLOW_INSTALL_EDITABLE_FLAG="" \
+ UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES}
+
+# Only copy install_mysql.sh to not invalidate cache on other script changes
+COPY scripts/docker/install_mysql.sh /scripts/docker/install_mysql.sh
+COPY docker-context-files /docker-context-files
Review comment:
Is there any reason why we copy these files in the middle of MySQL installation? It Is needed for Pypi only. Am I right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#issuecomment-823162386
[The Workflow run](https://github.com/apache/airflow/actions/runs/766716125) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#discussion_r616502878
##########
File path: Dockerfile
##########
@@ -139,98 +133,94 @@ RUN mkdir -pv /usr/share/man/man1 \
&& rm -rf /var/lib/apt/lists/*
ARG INSTALL_MYSQL_CLIENT="true"
-ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT}
-
-# Only copy install_mysql.sh to not invalidate cache on other script changes
-COPY scripts/docker/install_mysql.sh /scripts/docker/install_mysql.sh
-COPY docker-context-files /docker-context-files
-# fix permission issue in Azure DevOps when running the script
-RUN bash ./scripts/docker/install_mysql.sh dev
-
ARG AIRFLOW_REPO=apache/airflow
-ENV AIRFLOW_REPO=${AIRFLOW_REPO}
-
ARG AIRFLOW_BRANCH=master
-ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH}
-
ARG AIRFLOW_EXTRAS
ARG ADDITIONAL_AIRFLOW_EXTRAS=""
-ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
-
# Allows to override constraints source
ARG CONSTRAINTS_GITHUB_REPOSITORY="apache/airflow"
-ENV CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY}
-
ARG AIRFLOW_CONSTRAINTS="constraints"
-ENV AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS}
ARG AIRFLOW_CONSTRAINTS_REFERENCE=""
-ENV AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE}
ARG AIRFLOW_CONSTRAINTS_LOCATION=""
-ENV AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}
-
ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-master"
-ENV DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}
-
-ENV PATH=${PATH}:/root/.local/bin
-RUN mkdir -p /root/.local/bin
-
-RUN if [[ -f /docker-context-files/.pypirc ]]; then \
- cp /docker-context-files/.pypirc /root/.pypirc; \
- fi
-
ARG AIRFLOW_PIP_VERSION
-ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
-
# By default PIP has progress bar but you can disable it.
ARG PIP_PROGRESS_BAR
-ENV PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR}
-
-# Install Airflow with "--user" flag, so that we can copy the whole .local folder to the final image
-# from the build image and always in non-editable mode
-ENV AIRFLOW_INSTALL_USER_FLAG="--user"
-ENV AIRFLOW_INSTALL_EDITABLE_FLAG=""
-
-# Upgrade to specific PIP version
-RUN pip install --no-cache-dir --upgrade "pip==${AIRFLOW_PIP_VERSION}"
-
# By default we do not use pre-cached packages, but in CI/Breeze environment we override this to speed up
# builds in case setup.py/setup.cfg changed. This is pure optimisation of CI/Breeze builds.
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
-ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES}
-
+# This is airflow version that is put in the label of the image build
+ARG AIRFLOW_VERSION
# By default we install providers from PyPI but in case of Breeze build we want to install providers
# from local sources without the need of preparing provider packages upfront. This value is
# automatically overridden by Breeze scripts.
ARG INSTALL_PROVIDERS_FROM_SOURCES="false"
-ENV INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES}
-
-# This is airflow version that is put in the label of the image build
-ARG AIRFLOW_VERSION
-ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
-
# Determines the way airflow is installed. By default we install airflow from PyPI `apache-airflow` package
# But it also can be `.` from local installation or GitHub URL pointing to specific branch or tag
# Of Airflow. Note That for local source installation you need to have local sources of
# Airflow checked out together with the Dockerfile and AIRFLOW_SOURCES_FROM and AIRFLOW_SOURCES_TO
# set to "." and "/opt/airflow" respectively.
ARG AIRFLOW_INSTALLATION_METHOD="apache-airflow"
-ENV AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD}
-
# By default latest released version of airflow is installed (when empty) but this value can be overridden
# and we can install version according to specification (For example ==2.0.2 or <3.0.0).
ARG AIRFLOW_VERSION_SPECIFICATION=""
-ENV AIRFLOW_VERSION_SPECIFICATION=${AIRFLOW_VERSION_SPECIFICATION}
+# By default we do not upgrade to latest dependencies
+ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
+# By default we install latest airflow from PyPI so we do not need to copy sources of Airflow
+# but in case of breeze/CI builds we use latest sources and we override those
+# those SOURCES_FROM/TO with "." and "/opt/airflow" respectively
+ARG AIRFLOW_SOURCES_FROM="empty"
+ARG AIRFLOW_SOURCES_TO="/empty"
+
+
+ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \
+ AIRFLOW_REPO=${AIRFLOW_REPO} \
+ AIRFLOW_BRANCH=${AIRFLOW_BRANCH} \
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS} \
+ CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY} \
+ AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS} \
+ AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \
+ AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \
+ DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \
+ PATH=${PATH}:/root/.local/bin \
+ AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
+ PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR} \
+ # Install Airflow with "--user" flag, so that we can copy the whole .local folder to the final image
+ # from the build image and always in non-editable mode
+ AIRFLOW_INSTALL_USER_FLAG="--user" \
+ AIRFLOW_INSTALL_EDITABLE_FLAG="" \
+ UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES}
+
+# Only copy install_mysql.sh to not invalidate cache on other script changes
+COPY scripts/docker/install_mysql.sh /scripts/docker/install_mysql.sh
+COPY docker-context-files /docker-context-files
Review comment:
It can be moved few lines down. Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#discussion_r616597763
##########
File path: Dockerfile
##########
@@ -42,8 +42,6 @@ ARG AIRFLOW_HOME=/opt/airflow
ARG AIRFLOW_UID="50000"
ARG AIRFLOW_GID="50000"
-ARG CASS_DRIVER_BUILD_CONCURRENCY="8"
Review comment:
I kept this for "future" proofing, as cassandra-driver wasn't yet publishing py 3.9 wheels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#issuecomment-823352724
[The Workflow run](https://github.com/apache/airflow/actions/runs/767269151) is cancelling this PR. Building images for the PR has failed. Follow the workflow link to check the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #15438:
URL: https://github.com/apache/airflow/pull/15438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#discussion_r616266182
##########
File path: Dockerfile
##########
@@ -139,98 +133,94 @@ RUN mkdir -pv /usr/share/man/man1 \
&& rm -rf /var/lib/apt/lists/*
ARG INSTALL_MYSQL_CLIENT="true"
-ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT}
-
-# Only copy install_mysql.sh to not invalidate cache on other script changes
-COPY scripts/docker/install_mysql.sh /scripts/docker/install_mysql.sh
-COPY docker-context-files /docker-context-files
-# fix permission issue in Azure DevOps when running the script
-RUN bash ./scripts/docker/install_mysql.sh dev
-
ARG AIRFLOW_REPO=apache/airflow
-ENV AIRFLOW_REPO=${AIRFLOW_REPO}
-
ARG AIRFLOW_BRANCH=master
-ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH}
-
ARG AIRFLOW_EXTRAS
ARG ADDITIONAL_AIRFLOW_EXTRAS=""
-ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
-
# Allows to override constraints source
ARG CONSTRAINTS_GITHUB_REPOSITORY="apache/airflow"
-ENV CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY}
-
ARG AIRFLOW_CONSTRAINTS="constraints"
-ENV AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS}
ARG AIRFLOW_CONSTRAINTS_REFERENCE=""
-ENV AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE}
ARG AIRFLOW_CONSTRAINTS_LOCATION=""
-ENV AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}
-
ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-master"
-ENV DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}
-
-ENV PATH=${PATH}:/root/.local/bin
-RUN mkdir -p /root/.local/bin
-
-RUN if [[ -f /docker-context-files/.pypirc ]]; then \
- cp /docker-context-files/.pypirc /root/.pypirc; \
- fi
-
ARG AIRFLOW_PIP_VERSION
-ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
-
# By default PIP has progress bar but you can disable it.
ARG PIP_PROGRESS_BAR
-ENV PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR}
-
-# Install Airflow with "--user" flag, so that we can copy the whole .local folder to the final image
-# from the build image and always in non-editable mode
-ENV AIRFLOW_INSTALL_USER_FLAG="--user"
-ENV AIRFLOW_INSTALL_EDITABLE_FLAG=""
-
-# Upgrade to specific PIP version
-RUN pip install --no-cache-dir --upgrade "pip==${AIRFLOW_PIP_VERSION}"
-
# By default we do not use pre-cached packages, but in CI/Breeze environment we override this to speed up
# builds in case setup.py/setup.cfg changed. This is pure optimisation of CI/Breeze builds.
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
-ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES}
-
+# This is airflow version that is put in the label of the image build
+ARG AIRFLOW_VERSION
# By default we install providers from PyPI but in case of Breeze build we want to install providers
# from local sources without the need of preparing provider packages upfront. This value is
# automatically overridden by Breeze scripts.
ARG INSTALL_PROVIDERS_FROM_SOURCES="false"
-ENV INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES}
-
-# This is airflow version that is put in the label of the image build
-ARG AIRFLOW_VERSION
-ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
-
# Determines the way airflow is installed. By default we install airflow from PyPI `apache-airflow` package
# But it also can be `.` from local installation or GitHub URL pointing to specific branch or tag
# Of Airflow. Note That for local source installation you need to have local sources of
# Airflow checked out together with the Dockerfile and AIRFLOW_SOURCES_FROM and AIRFLOW_SOURCES_TO
# set to "." and "/opt/airflow" respectively.
ARG AIRFLOW_INSTALLATION_METHOD="apache-airflow"
-ENV AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD}
-
# By default latest released version of airflow is installed (when empty) but this value can be overridden
# and we can install version according to specification (For example ==2.0.2 or <3.0.0).
ARG AIRFLOW_VERSION_SPECIFICATION=""
-ENV AIRFLOW_VERSION_SPECIFICATION=${AIRFLOW_VERSION_SPECIFICATION}
+# By default we do not upgrade to latest dependencies
+ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
+# By default we install latest airflow from PyPI so we do not need to copy sources of Airflow
+# but in case of breeze/CI builds we use latest sources and we override those
+# those SOURCES_FROM/TO with "." and "/opt/airflow" respectively
+ARG AIRFLOW_SOURCES_FROM="empty"
+ARG AIRFLOW_SOURCES_TO="/empty"
+
+
+ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \
+ AIRFLOW_REPO=${AIRFLOW_REPO} \
+ AIRFLOW_BRANCH=${AIRFLOW_BRANCH} \
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS} \
+ CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY} \
+ AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS} \
+ AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \
+ AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \
+ DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \
+ PATH=${PATH}:/root/.local/bin \
+ AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
+ PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR} \
+ # Install Airflow with "--user" flag, so that we can copy the whole .local folder to the final image
+ # from the build image and always in non-editable mode
+ AIRFLOW_INSTALL_USER_FLAG="--user" \
+ AIRFLOW_INSTALL_EDITABLE_FLAG="" \
+ UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES}
+
+# Only copy install_mysql.sh to not invalidate cache on other script changes
+COPY scripts/docker/install_mysql.sh /scripts/docker/install_mysql.sh
+COPY docker-context-files /docker-context-files
Review comment:
Is there any reason why we copy these files in the middle of MySQL installation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#discussion_r616674243
##########
File path: Dockerfile
##########
@@ -42,8 +42,6 @@ ARG AIRFLOW_HOME=/opt/airflow
ARG AIRFLOW_UID="50000"
ARG AIRFLOW_GID="50000"
-ARG CASS_DRIVER_BUILD_CONCURRENCY="8"
Review comment:
Ah. True. Will restore it back. We might soon add python 3.9 finally (waiting for newest beam release).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#issuecomment-822888737
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#issuecomment-822718098
[The Workflow run](https://github.com/apache/airflow/actions/runs/764568638) is cancelling this PR. Building images for the PR has failed. Follow the workflow link to check the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15438: Speeds up Docker build process by combining RUN and ENV commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15438:
URL: https://github.com/apache/airflow/pull/15438#issuecomment-823471868
Triggering re-build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org