You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/17 09:39:35 UTC
[GitHub] [airflow] ecerulm opened a new issue #14845: Dockerfile fails to build with default set of extras
ecerulm opened a new issue #14845:
URL: https://github.com/apache/airflow/issues/14845
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: master
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
Trying to build the docker image for 2.0.1 with default extras:
```
git clone --depth 1 https://github.com/apache/airflow.git airflow-repo
cd airflow-repo
git log -1 # 03d3c7db931df34a724a3f254bf13197a9063576 Kamil Breguła Wed Mar 17 07:15:18 2021 +0100
docker build . \
--tag baseairflow:2.0.1 \
--build-arg AIRFLOW_VERSION="2.0.1" \
--build-arg AIRFLOW_INSTALL_VERSION="==2.0.1" \
--build-arg PYTHON_BASE_IMAGE='python:3.8-buster' \
--build-arg PYTHON_MAJOR_MINOR_VERSION='3.8' \
2>&1 | tee ../docker-baseairflow.log
```
fails with
```
#26 147.5 Attempting uninstall: google-cloud-bigquery-storage
#26 147.5 Found existing installation: google-cloud-bigquery-storage 2.3.0
#26 147.5 Uninstalling google-cloud-bigquery-storage-2.3.0:
#26 147.9 Successfully uninstalled google-cloud-bigquery-storage-2.3.0
#26 148.1 ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.
#26 148.1
#26 148.1 We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.
#26 148.1
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-automl<3.0.0,>=2.1.0, but you'll have google-cloud-automl 1.0.1 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-bigquery-datatransfer<4.0.0,>=3.0.0, but you'll have google-cloud-bigquery-datatransfer 1.1.1 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-datacatalog<4.0.0,>=3.0.0, but you'll have google-cloud-datacatalog 0.7.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-dataproc<3.0.0,>=2.2.0, but you'll have google-cloud-dataproc 1.1.1 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-kms<3.0.0,>=2.0.0, but you'll have google-cloud-kms 1.4.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-logging<3.0.0,>=2.1.1, but you'll have google-cloud-logging 1.15.1 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-monitoring<3.0.0,>=2.0.0, but you'll have google-cloud-monitoring 1.1.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-os-login<3.0.0,>=2.0.0, but you'll have google-cloud-os-login 1.0.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-pubsub<3.0.0,>=2.0.0, but you'll have google-cloud-pubsub 1.7.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-redis<3.0.0,>=2.0.0, but you'll have google-cloud-redis 1.0.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-tasks<3.0.0,>=2.0.0, but you'll have google-cloud-tasks 1.5.0 which is incompatible.
#26 148.1 Successfully installed flask-appbuilder-3.1.1 google-cloud-automl-1.0.1 google-cloud-bigquery-datatransfer-1.1.1 google-cloud-bigquery-storage-1.1.0 google-cloud-datacatalog-0.7.0 google-cloud-dataproc-1.1.1 google-cloud-kms-1.4.0 google-cloud-logging-1.15.1 google-cloud-monitoring-1.1.0 google-cloud-os-login-1.0.0 google-cloud-pubsub-1.7.0 google-cloud-redis-1.0.0 google-cloud-tasks-1.5.0 pyarrow-1.0.1
#26 148.3 WARNING: You are using pip version 20.2.4; however, version 21.0.1 is available.
#26 148.3 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
#26 148.4 + pip install --user --upgrade pip==20.2.4
#26 149.3 Requirement already up-to-date: pip==20.2.4 in /usr/local/lib/python3.8/site-packages (20.2.4)
#26 155.6 WARNING: You are using pip version 20.2.4; however, version 21.0.1 is available.
#26 155.6 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
#26 155.7 + pip check
#26 162.0 apache-airflow-providers-google 2.1.0 has requirement google-cloud-automl<3.0.0,>=2.1.0, but you have google-cloud-automl 1.0.1.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-bigquery-datatransfer<4.0.0,>=3.0.0, but you have google-cloud-bigquery-datatransfer 1.1.1.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-datacatalog<4.0.0,>=3.0.0, but you have google-cloud-datacatalog 0.7.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-dataproc<3.0.0,>=2.2.0, but you have google-cloud-dataproc 1.1.1.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-kms<3.0.0,>=2.0.0, but you have google-cloud-kms 1.4.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-logging<3.0.0,>=2.1.1, but you have google-cloud-logging 1.15.1.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-monitoring<3.0.0,>=2.0.0, but you have google-cloud-monitoring 1.1.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-os-login<3.0.0,>=2.0.0, but you have google-cloud-os-login 1.0.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-pubsub<3.0.0,>=2.0.0, but you have google-cloud-pubsub 1.7.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-redis<3.0.0,>=2.0.0, but you have google-cloud-redis 1.0.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-tasks<3.0.0,>=2.0.0, but you have google-cloud-tasks 1.5.0.
#26 162.1 + false
#26 ERROR: executor failed running [/bin/bash -o pipefail -e -u -x -c if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true]: exit code: 1
------
> [airflow-build-image 15/18] RUN if [[ true == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if [[ -n "" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true:
------
executor failed running [/bin/bash -o pipefail -e -u -x -c if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || tr
```
**What you expected to happen**:
I expected to build the image,
**How to reproduce it**:
```
git clone --depth 1 https://github.com/apache/airflow.git airflow-repo
cd airflow-repo
git log -1 # 03d3c7db931df34a724a3f254bf13197a9063576 Kamil Breguła Wed Mar 17 07:15:18 2021 +0100
docker build . \
--tag baseairflow:2.0.1 \
--build-arg AIRFLOW_VERSION="2.0.1" \
--build-arg AIRFLOW_INSTALL_VERSION="==2.0.1" \
--build-arg PYTHON_BASE_IMAGE='python:3.8-buster' \
--build-arg PYTHON_MAJOR_MINOR_VERSION='3.8' \
2>&1 | tee ../docker-baseairflow.log
```
**Anything else we need to know**:
If instead of using the default extras `"async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv"` I narrow it down to `'async,cncf.kubernetes,microsoft.mssql'` then it will build successfully
```
docker build . \
--tag baseairflow:2.0.1 \
--build-arg AIRFLOW_VERSION="2.0.1" \
--build-arg AIRFLOW_INSTALL_VERSION="==2.0.1" \
--build-arg PYTHON_BASE_IMAGE='python:3.8-buster' \
--build-arg PYTHON_MAJOR_MINOR_VERSION='3.8' \
--build-arg AIRFLOW_EXTRAS='async,cncf.kubernetes,microsoft.mssql' \
2>&1 | tee ../docker-baseairflow.log
```
So I'm guessing that some of the extras is causing the issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm commented on issue #14845: Dockerfile fails to build with default set of extras
Posted by GitBox <gi...@apache.org>.
ecerulm commented on issue #14845:
URL: https://github.com/apache/airflow/issues/14845#issuecomment-800992940
After conversation with @potiuk, the problem is that it will take the constraints from master, you need to make sure that it takes the constraints for 2.0.1 (with `AIRFLOW_CONSTRAINTS_REFERENCE` ) so that the right versions of the provider packages are installed:
```
docker build . \
--tag baseairflow:2.0.1 \
--build-arg AIRFLOW_VERSION="2.0.1" \
--build-arg AIRFLOW_INSTALL_VERSION="==2.0.1" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2.0.1"
--build-arg PYTHON_BASE_IMAGE='python:3.8-buster' \
--build-arg PYTHON_MAJOR_MINOR_VERSION='3.8' \
2>&1 | tee ../docker-baseairflow.log
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm closed issue #14845: Dockerfile fails to build with default set of extras
Posted by GitBox <gi...@apache.org>.
ecerulm closed issue #14845:
URL: https://github.com/apache/airflow/issues/14845
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm commented on issue #14845: Dockerfile fails to build with default set of extras
Posted by GitBox <gi...@apache.org>.
ecerulm commented on issue #14845:
URL: https://github.com/apache/airflow/issues/14845#issuecomment-800960715
I think the `google` extra is the one causing the issue. I took the default AIRFLOW_EXTRAS (`"async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv"`) and removed `google` from it :
```
--build-arg AIRFLOW_EXTRAS="async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv"
```
and this builds
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org