You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2021/03/28 01:16:57 UTC
[airflow] 02/02: Adds dill exclusion to Dockerfiles to accomodate
upcoming beam fix (#15048)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c25ae8198647747e881fe69f9f2efbcaaa166135
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Mar 27 23:41:53 2021 +0100
Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix (#15048)
* Upgrades moto to newer version (~=2.0)
According to https://github.com/spulec/moto/issues/3535#issuecomment-808706939
1.3.17 version of moto with a fix to be compatible with mock> 4.0.3 is
not going to be released because of breaking changes. Therefore we need
to migrate to newer version of moto.
At the same time we can get rid of the old botocore limitation, which
was added apparently to handle some test errors. We are relying fully
on what boto3 depends on.
Upgrading dependencies also discovered that mysql tests need to
be fixed because upgraded version of dependencies cause some test
failure (those turned out to be badly written tests).
* Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix
With the upcoming apache-beam change where mock library will be
removed from install dependencies, we will be able to remove
`apache-beam` exclusion in our CI scripts. This will be a final
step of cleaning dependencies so that we have a truly
golden set of constraints that will allow to install airflow
and all community managed providers (we managed to fix all those
dependency issues for all packages but apache-beam).
The fix https://github.com/apache/beam/pull/14328 when merged
and Apache Beam is released will allow us to migrate to the new
version and get rid of the CI exclusion for beam.
Closes: #14994
(cherry picked from commit ec962b01b72f76d6d2193342e52110996246a45a)
---
Dockerfile | 3 ++-
Dockerfile.ci | 3 ++-
setup.py | 7 +++++--
3 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index 2d4aad7..3928057 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -276,7 +276,8 @@ ENV INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI}
# * chardet<4 - required to keep snowflake happy
# * urllib3 - required to keep boto3 happy
# * pyjwt<2.0.0: flask-jwt-extended requires it
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0"
+# * dill<0.3.3 required by apache-beam
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3"
WORKDIR /opt/airflow
diff --git a/Dockerfile.ci b/Dockerfile.ci
index ad72d70..692a084 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -348,7 +348,8 @@ COPY airflow/__init__.py ${AIRFLOW_SOURCES}/airflow/__init__.py
# * pyOpenSSL: required by snowflake provider https://github.com/snowflakedb/snowflake-connector-python/blob/v2.3.6/setup.py#L201
# * urllib3<1.26: Required to keep boto3 happy
# * pyjwt<2.0.0: flask-jwt-extended requires it
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 lazy-object-proxy<1.5.0 pyOpenSSL<20.0.0 urllib3<1.26 pyjwt<2.0.0"
+# * dill<0.3.3 required by apache-beam
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 lazy-object-proxy<1.5.0 pyOpenSSL<20.0.0 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3"
ENV EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
ARG CONTINUE_ON_PIP_CHECK_FAILURE="false"
diff --git a/setup.py b/setup.py
index 26ad19a..0f421bc 100644
--- a/setup.py
+++ b/setup.py
@@ -742,8 +742,11 @@ PACKAGES_EXCLUDED_FOR_ALL.extend(
# This can be removed as soon as we get non-conflicting
# requirements for the apache-beam as well.
#
-# Currently Apache Beam has very narrow and old dependencies for 'dill' and 'mock' packages which
-# are required by our tests (but only for tests).
+# Currently Apache Beam has very narrow and old dependencies for 'mock' package which
+# are required only for our tests.
+# once https://github.com/apache/beam/pull/14328 is solved and new version of apache-beam is released
+# we will be able to remove this exclusion and get rid of `install_remaining_dependencies`
+# function in `scripts/in_container`.
#
PACKAGES_EXCLUDED_FOR_CI = [
'apache-beam',