You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2023/07/21 17:27:58 UTC
[airflow] branch main updated: Fix ARM image building after Cython 3.0.0 release (#32748)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 8e054bb57d Fix ARM image building after Cython 3.0.0 release (#32748)
8e054bb57d is described below
commit 8e054bb57dd0b29f302872b4500b799c369bde71
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Jul 21 19:27:51 2023 +0200
Fix ARM image building after Cython 3.0.0 release (#32748)
Workaround an issue with installing pymssql on ARM architecture
triggered by Cython 3.0.0 release as of 18 July 2023. The problem is
that pip uses latest Cython to compile pymssql and since we are using
setuptools, there is no easy way to fix version of Cython used to
compile packages.
This triggers a problem with newer `pip` versions that have build
isolation enabled by default because There is no (easy) way to pin build
dependencies for dependent packages. If a package does not have limit
on build dependencies, it will use the latest version of them to build
that particular package.
The workaround to the problem suggest in the last thread by Pradyun
Gedam - pip maintainer - is to use PIP_CONSTRAINT environment variable
and constraint the version of Cython used while installing
the package. Which is precisely what we are doing here.
Note that it does not work if we pass ``--constraint`` option to pip
because it will not be passed to the package being build in isolation.
The fact that the PIP_CONSTRAINT env variable works in the isolation is
a bit of side-effect on how env variables work and that they are passed
to subprocesses as pip launches a subprocess `pip` to build the package.
This is a temporary solution until the issue is resolved in pymssql or
Cython.
Issues/discussions that track it:
* https://github.com/cython/cython/issues/5541
* https://github.com/pymssql/pymssql/pull/827
* https://discuss.python.org/t/no-way-to-pin-build-dependencies/29833
Since we have to change Dockerfile around installing `pip`, also
version of `pip` has been upgraded to latest - 23.2
---
Dockerfile | 59 +++++++++++++++++++++--
Dockerfile.ci | 50 +++++++++++++++++--
IMAGES.rst | 2 +-
dev/breeze/src/airflow_breeze/global_constants.py | 2 +-
docs/docker-stack/build-arg-ref.rst | 2 +-
scripts/docker/common.sh | 2 +-
scripts/docker/install_mssql.sh | 41 ++++++++++++++++
7 files changed, 147 insertions(+), 11 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index 3f08074259..14f02ec903 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -48,7 +48,7 @@ ARG AIRFLOW_VERSION="2.6.3"
ARG PYTHON_BASE_IMAGE="python:3.8-slim-bullseye"
-ARG AIRFLOW_PIP_VERSION=23.1.2
+ARG AIRFLOW_PIP_VERSION=23.2
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
ARG AIRFLOW_IMAGE_README_URL="https://raw.githubusercontent.com/apache/airflow/main/docs/docker-stack/README.md"
@@ -266,6 +266,10 @@ EOF
COPY <<"EOF" /install_mssql.sh
set -euo pipefail
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
: "${INSTALL_MSSQL_CLIENT:?Should be true or false}"
COLOR_BLUE=$'\e[34m'
@@ -298,6 +302,40 @@ function install_mssql_client() {
rm -rf /var/lib/apt/lists/*
apt-get autoremove -yqq --purge
apt-get clean && rm -rf /var/lib/apt/lists/*
+
+ # Workaround an issue with installing pymssql on ARM architecture triggered by Cython 3.0.0 release as of
+ # 18 July 2023. The problem is that pip uses latest Cython to compile pymssql and since we are using
+ # setuptools, there is no easy way to fix version of Cython used to compile packages.
+ #
+ # This triggers a problem with newer `pip` versions that have build isolation enabled by default because
+ # There is no (easy) way to pin build dependencies for dependent packages. If a package does not have
+ # limit on build dependencies, it will use the latest version of them to build that particular package.
+ #
+ # The workaround to the problem suggest in the last thread by Pradyun Gedam - pip maintainer - is to
+ # use PIP_CONSTRAINT environment variable and constraint the version of Cython used while installing
+ # the package. Which is precisely what we are doing here.
+ #
+ # Note that it does not work if we pass ``--constraint`` option to pip because it will not be passed to
+ # the package being build in isolation. The fact that the PIP_CONSTRAINT env variable works in the isolation
+ # is a bit of side-effect on how env variables work and that they are passed to subprocesses as pip
+ # launches a subprocess `pip` to build the package.
+ #
+ # This is a temporary solution until the issue is resolved in pymssql or Cython
+ # Issues/discussions that track it:
+ #
+ # * https://github.com/cython/cython/issues/5541
+ # * https://github.com/pymssql/pymssql/pull/827
+ # * https://discuss.python.org/t/no-way-to-pin-build-dependencies/29833
+ #
+ # TODO: Remove this workaround when the issue is resolved.
+ # ALSO REMOVE THE TOP LINES ABOVE WITH common.sh IMPORT AS WELL AS COPYING common.sh ib
+ # Dockerfile AND Dockerfile.ci (look for capital PYMSSQL - there are several places to remove)
+ if [[ "${1}" == "dev" ]]; then
+ common::install_pip_version
+ echo "Cython==0.29.36" >> /tmp/mssql-constraints.txt
+ PIP_CONSTRAINT=/tmp/mssql-constraints.txt pip install pymssql
+ rm /tmp/mssql-constraints.txt
+ fi
}
install_mssql_client "${@}"
@@ -433,7 +471,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="23.1.2"
+ export AIRFLOW_PIP_VERSION="23.2"
fi
fi
}
@@ -1157,8 +1195,15 @@ ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \
# scripts which are needed much later will not invalidate the docker layer here
COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/
+# THE 3 LINES ARE ONLY NEEDED IN ORDER TO MAKE PYMSSQL BUILD WORK WITH LATEST CYTHON
+# AND SHOULD BE REMOVED WHEN WORKAROUND IN install_mssql.sh IS REMOVED
+ARG AIRFLOW_PIP_VERSION=23.2
+ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
+COPY --from=scripts common.sh /scripts/docker/
+
+
RUN bash /scripts/docker/install_mysql.sh dev && \
- bash /scripts/docker/install_mssql.sh && \
+ bash /scripts/docker/install_mssql.sh dev && \
bash /scripts/docker/install_postgres.sh dev
ENV PATH=${PATH}:/opt/mssql-tools/bin
@@ -1340,6 +1385,12 @@ ENV PATH="${AIRFLOW_USER_HOME_DIR}/.local/bin:${PATH}" \
AIRFLOW_USER_HOME_DIR=${AIRFLOW_USER_HOME_DIR} \
AIRFLOW_HOME=${AIRFLOW_HOME}
+# THE 3 LINES ARE ONLY NEEDED IN ORDER TO MAKE PYMSSQL BUILD WORK WITH LATEST CYTHON
+# AND SHOULD BE REMOVED WHEN WORKAROUND IN install_mssql.sh IS REMOVED
+ARG AIRFLOW_PIP_VERSION=23.2
+ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
+COPY --from=scripts common.sh /scripts/docker/
+
# Only copy mysql/mssql installation scripts for now - so that changing the other
# scripts which are needed much later will not invalidate the docker layer here.
COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/
@@ -1348,7 +1399,7 @@ COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scrip
# had different umask set and group x bit was not set. In Azure the bit might be not set at all.
# That also protects against AUFS Docker backend problem where changing the executable bit required sync
RUN bash /scripts/docker/install_mysql.sh prod \
- && bash /scripts/docker/install_mssql.sh \
+ && bash /scripts/docker/install_mssql.sh prod \
&& bash /scripts/docker/install_postgres.sh prod \
&& adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password \
--quiet "airflow" --uid "${AIRFLOW_UID}" --gid "0" --home "${AIRFLOW_USER_HOME_DIR}" \
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 23a24d26dd..bbea500d8f 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -226,6 +226,10 @@ EOF
COPY <<"EOF" /install_mssql.sh
set -euo pipefail
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
: "${INSTALL_MSSQL_CLIENT:?Should be true or false}"
COLOR_BLUE=$'\e[34m'
@@ -258,6 +262,40 @@ function install_mssql_client() {
rm -rf /var/lib/apt/lists/*
apt-get autoremove -yqq --purge
apt-get clean && rm -rf /var/lib/apt/lists/*
+
+ # Workaround an issue with installing pymssql on ARM architecture triggered by Cython 3.0.0 release as of
+ # 18 July 2023. The problem is that pip uses latest Cython to compile pymssql and since we are using
+ # setuptools, there is no easy way to fix version of Cython used to compile packages.
+ #
+ # This triggers a problem with newer `pip` versions that have build isolation enabled by default because
+ # There is no (easy) way to pin build dependencies for dependent packages. If a package does not have
+ # limit on build dependencies, it will use the latest version of them to build that particular package.
+ #
+ # The workaround to the problem suggest in the last thread by Pradyun Gedam - pip maintainer - is to
+ # use PIP_CONSTRAINT environment variable and constraint the version of Cython used while installing
+ # the package. Which is precisely what we are doing here.
+ #
+ # Note that it does not work if we pass ``--constraint`` option to pip because it will not be passed to
+ # the package being build in isolation. The fact that the PIP_CONSTRAINT env variable works in the isolation
+ # is a bit of side-effect on how env variables work and that they are passed to subprocesses as pip
+ # launches a subprocess `pip` to build the package.
+ #
+ # This is a temporary solution until the issue is resolved in pymssql or Cython
+ # Issues/discussions that track it:
+ #
+ # * https://github.com/cython/cython/issues/5541
+ # * https://github.com/pymssql/pymssql/pull/827
+ # * https://discuss.python.org/t/no-way-to-pin-build-dependencies/29833
+ #
+ # TODO: Remove this workaround when the issue is resolved.
+ # ALSO REMOVE THE TOP LINES ABOVE WITH common.sh IMPORT AS WELL AS COPYING common.sh ib
+ # Dockerfile AND Dockerfile.ci (look for capital PYMSSQL - there are several places to remove)
+ if [[ "${1}" == "dev" ]]; then
+ common::install_pip_version
+ echo "Cython==0.29.36" >> /tmp/mssql-constraints.txt
+ PIP_CONSTRAINT=/tmp/mssql-constraints.txt pip install pymssql
+ rm /tmp/mssql-constraints.txt
+ fi
}
install_mssql_client "${@}"
@@ -393,7 +431,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="23.1.2"
+ export AIRFLOW_PIP_VERSION="23.2"
fi
fi
}
@@ -1222,6 +1260,12 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
COPY --from=scripts install_os_dependencies.sh /scripts/docker/
RUN bash /scripts/docker/install_os_dependencies.sh dev
+# THE 3 LINES ARE ONLY NEEDED IN ORDER TO MAKE PYMSSQL BUILD WORK WITH LATEST CYTHON
+# AND SHOULD BE REMOVED WHEN WORKAROUND IN install_mssql.sh IS REMOVED
+ARG AIRFLOW_PIP_VERSION=23.2
+ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
+COPY --from=scripts common.sh /scripts/docker/
+
# Only copy mysql/mssql installation scripts for now - so that changing the other
# scripts which are needed much later will not invalidate the docker layer here.
COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/
@@ -1240,7 +1284,7 @@ ENV HOME=${HOME} \
# That also protects against AUFS Docker backen dproblem where changing the executable bit required sync
RUN bash /scripts/docker/install_mysql.sh prod \
&& bash /scripts/docker/install_mysql.sh dev \
- && bash /scripts/docker/install_mssql.sh \
+ && bash /scripts/docker/install_mssql.sh dev \
&& bash /scripts/docker/install_postgres.sh dev
# Install Helm
@@ -1275,7 +1319,7 @@ ARG AIRFLOW_CI_BUILD_EPOCH="4"0
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
# By default in the image, we are installing all providers when installing from sources
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
-ARG AIRFLOW_PIP_VERSION=23.1.2
+ARG AIRFLOW_PIP_VERSION=23.2
# Setup PIP
# By default PIP install run without cache to make image smaller
ARG PIP_NO_CACHE_DIR="true"
diff --git a/IMAGES.rst b/IMAGES.rst
index cb049bf005..2fa95b4a9c 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -462,7 +462,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| ``ADDITIONAL_DEV_APT_ENV`` | | Additional env variables defined |
| | | when installing dev deps |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``AIRFLOW_PIP_VERSION`` | ``23.1.2`` | PIP version used. |
+| ``AIRFLOW_PIP_VERSION`` | ``23.2`` | PIP version used. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``PIP_PROGRESS_BAR`` | ``on`` | Progress bar for PIP installation |
+------------------------------------------+------------------------------------------+------------------------------------------+
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index 7b79af5ad4..18ff5d7828 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -87,7 +87,7 @@ ALLOWED_POSTGRES_VERSIONS = ["11", "12", "13", "14", "15"]
ALLOWED_MYSQL_VERSIONS = ["5.7", "8"]
ALLOWED_MSSQL_VERSIONS = ["2017-latest", "2019-latest"]
-PIP_VERSION = "23.1.2"
+PIP_VERSION = "23.2"
@lru_cache(maxsize=None)
diff --git a/docs/docker-stack/build-arg-ref.rst b/docs/docker-stack/build-arg-ref.rst
index 6da9cdf145..8544beff6d 100644
--- a/docs/docker-stack/build-arg-ref.rst
+++ b/docs/docker-stack/build-arg-ref.rst
@@ -45,7 +45,7 @@ Those are the most common arguments that you use when you want to build a custom
+------------------------------------------+------------------------------------------+---------------------------------------------+
| ``AIRFLOW_USER_HOME_DIR`` | ``/home/airflow`` | Home directory of the Airflow user. |
+------------------------------------------+------------------------------------------+---------------------------------------------+
-| ``AIRFLOW_PIP_VERSION`` | ``23.1.2`` | PIP version used. |
+| ``AIRFLOW_PIP_VERSION`` | ``23.2`` | PIP version used. |
+------------------------------------------+------------------------------------------+---------------------------------------------+
| ``ADDITIONAL_PIP_INSTALL_FLAGS`` | | additional ``pip`` flags passed to the |
| | | installation commands (except when |
diff --git a/scripts/docker/common.sh b/scripts/docker/common.sh
index ba04841b7a..f6066c3806 100644
--- a/scripts/docker/common.sh
+++ b/scripts/docker/common.sh
@@ -42,7 +42,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="23.1.2"
+ export AIRFLOW_PIP_VERSION="23.2"
fi
fi
}
diff --git a/scripts/docker/install_mssql.sh b/scripts/docker/install_mssql.sh
index fa0cfc0033..883819a225 100644
--- a/scripts/docker/install_mssql.sh
+++ b/scripts/docker/install_mssql.sh
@@ -17,6 +17,13 @@
# shellcheck shell=bash
set -euo pipefail
+# REMOVE THOSE 3 LINES BELOW TOGETHER WITH PYMSSQL INSTALLATION BELOW AFTER PYMSSQL
+# INSTALLATION WITH CYTHON 3.0.0 IS FIXED SEE BELOW FOR MORE DETAILS
+# shellcheck source=scripts/docker/common.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
: "${INSTALL_MSSQL_CLIENT:?Should be true or false}"
COLOR_BLUE=$'\e[34m'
@@ -49,6 +56,40 @@ function install_mssql_client() {
rm -rf /var/lib/apt/lists/*
apt-get autoremove -yqq --purge
apt-get clean && rm -rf /var/lib/apt/lists/*
+
+ # Workaround an issue with installing pymssql on ARM architecture triggered by Cython 3.0.0 release as of
+ # 18 July 2023. The problem is that pip uses latest Cython to compile pymssql and since we are using
+ # setuptools, there is no easy way to fix version of Cython used to compile packages.
+ #
+ # This triggers a problem with newer `pip` versions that have build isolation enabled by default because
+ # There is no (easy) way to pin build dependencies for dependent packages. If a package does not have
+ # limit on build dependencies, it will use the latest version of them to build that particular package.
+ #
+ # The workaround to the problem suggest in the last thread by Pradyun Gedam - pip maintainer - is to
+ # use PIP_CONSTRAINT environment variable and constraint the version of Cython used while installing
+ # the package. Which is precisely what we are doing here.
+ #
+ # Note that it does not work if we pass ``--constraint`` option to pip because it will not be passed to
+ # the package being build in isolation. The fact that the PIP_CONSTRAINT env variable works in the isolation
+ # is a bit of side-effect on how env variables work and that they are passed to subprocesses as pip
+ # launches a subprocess `pip` to build the package.
+ #
+ # This is a temporary solution until the issue is resolved in pymssql or Cython
+ # Issues/discussions that track it:
+ #
+ # * https://github.com/cython/cython/issues/5541
+ # * https://github.com/pymssql/pymssql/pull/827
+ # * https://discuss.python.org/t/no-way-to-pin-build-dependencies/29833
+ #
+ # TODO: Remove this workaround when the issue is resolved.
+ # ALSO REMOVE THE TOP LINES ABOVE WITH common.sh IMPORT AS WELL AS COPYING common.sh ib
+ # Dockerfile AND Dockerfile.ci (look for capital PYMSSQL - there are several places to remove)
+ if [[ "${1}" == "dev" ]]; then
+ common::install_pip_version
+ echo "Cython==0.29.36" >> /tmp/mssql-constraints.txt
+ PIP_CONSTRAINT=/tmp/mssql-constraints.txt pip install pymssql
+ rm /tmp/mssql-constraints.txt
+ fi
}
install_mssql_client "${@}"