You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/06 16:33:32 UTC
[GitHub] [airflow] jackemuk-es opened a new issue #15231: New version of docutils breaks Dockerfile for tag 2.0.1 (branch v2-0-stable)
jackemuk-es opened a new issue #15231:
URL: https://github.com/apache/airflow/issues/15231
**Apache Airflow version**:
2.0.1
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
None
**Environment**:
- **Cloud provider or hardware configuration**: aws, t3.medium
- **OS** (e.g. from /etc/os-release): Ubuntu 20.04.2 LTS (Focal Fossa)
- **Kernel** (e.g. `uname -a`): Linux ip-10-0-2-6 5.4.0-1041-aws #43-Ubuntu SMP Fri Mar 19 22:06:16 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
**What happened**:
Building custom airflow image was working before docutils==0.17 was released on April 3rd, 2021 using the docker-compose file from https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml
$> docker-compose build
...
...
...
Building airflow-init
Step 1/153 : ARG AIRFLOW_VERSION="2.0.0.dev0"
Step 2/153 : ...
...
...
...
Step 84/153 : RUN if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true
---> Running in c2bceecda1cd
+ [[ true == \t\r\u\e ]]
+ bash /scripts/docker/install_airflow.sh
...
...
...
Installing collected packages: pytz, Babel, idna, dnspython, email-validator, itsdangerous, MarkupSafe, Jinja2, Werkzeug, click, Flask, Flask-Login, colorama, SQLAlchemy, defusedxml, python3-openid, Flask-OpenID, WTForms, Flask-WTF, six, python-dateutil, marshmallow, marshmallow-sqlalchemy, pyrsistent, attrs, jsonschema, marshmallow-enum, PyYAML, apispec, prison, Flask-SQLAlchemy, PyJWT, Flask-JWT-Extended, Flask-Babel, SQLAlchemy-Utils, Flask-AppBuilder, Flask-Caching, Mako, Markdown, pycparser, cffi, PyNaCl, Pygments, SQLAlchemy-JSONField, sphinxcontrib-qthelp, sphinxcontrib-serializinghtml, sphinxcontrib-applehelp, docutils, sphinxcontrib-htmlhelp, pyparsing, packaging, imagesize, alabaster, sphinxcontrib-jsmath, snowballstemmer, sphinxcontrib-devhelp, urllib3, chardet, certifi, requests, Sphinx, Unidecode, cryptography, adal, multidict, yarl, typing-extensions, async-timeout, aiohttp, python-editor, alembic, vine, amqp, textwrap3, ansiwrap, graphviz, importlib-resources, iso86
01, lazy-object-proxy, text-unidecode, python-slugify, python-nvd3, unicodecsv, apache-airflow-providers-ftp, tenacity, lockfile, python-daemon, marshmallow-oneofschema, tabulate, colorlog, psutil, numpy, pandas, argcomplete, termcolor, apache-airflow-providers-sqlite, dill, apache-airflow-providers-imap, zipp, importlib-metadata, pytzdata, pendulum, commonmark, rich, cached-property, setproctitle, cattrs, apache-airflow-providers-http, gunicorn, natsort, croniter, isodate, openapi-schema-validator, openapi-spec-validator, clickclick, inflection, swagger-ui-bundle, connexion, jmespath, botocore, s3transfer, boto3, watchtower, tornado, kombu, billiard, celery, prometheus-client, humanize, flower, apache-airflow-providers-celery, sphinxcontrib-dotnetdomain, wrapt, astroid, sphinxcontrib-golangdomain, sphinx-autoapi, pyenchant, sphinxcontrib-spelling, sphinx-airflow-theme, sphinx-rtd-theme, sphinx-argparse, sphinx-jinja, sphinx-copybutton, sphinxcontrib-httpdomain, sphinxcontrib-redoc,
websocket-client, docker-pycreds, docker, apache-airflow-providers-docker, elasticsearch, elasticsearch-dbapi, elasticsearch-dsl, apache-airflow-providers-elasticsearch, grpcio, httplib2, pyasn1, pyasn1-modules, rsa, cachetools, google-auth, google-auth-httplib2, apache-airflow-providers-grpc, hvac, apache-airflow-providers-hashicorp, oauthlib, requests-oauthlib, requests-toolbelt, pbr, jira, apache-airflow-providers-jira, kubernetes, apache-airflow-providers-cncf-kubernetes, apache-airflow-providers-openfaas, apache-airflow-providers-opsgenie, cx-Oracle, apache-airflow-providers-oracle, pdpyras, apache-airflow-providers-pagerduty, mypy-extensions, toml, typed-ast, appdirs, regex, pathspec, black, tqdm, nest-asyncio, async-generator, ipython-genutils, traitlets, jupyter-core, pyzmq, jupyter-client, nbformat, nbclient, entrypoints, fsspec, google-auth-oauthlib, ujson, decorator, gcsfs, pyarrow, azure-datalake-store, msrest, azure-core, azure-storage-blob, papermill, parso, jedi, wcw
idth, prompt-toolkit, backcall, ptyprocess, pexpect, pickleshare, ipython, nteract-scrapbook, apache-airflow-providers-papermill, psycopg2-binary, apache-airflow-providers-postgres, redis, apache-airflow-providers-redis, bcrypt, paramiko, pysftp, sshtunnel, apache-airflow-providers-ssh, apache-airflow-providers-sftp, slack-sdk, apache-airflow-providers-slack, slackclient, apache-airflow, apache-airflow-providers-amazon
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.
We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.
sphinx-rtd-theme 0.5.2 requires docutils<0.17, but you'll have docutils 0.17 which is incompatible.
apache-airflow 2.0.1 requires flask-appbuilder~=3.1.1, but you'll have flask-appbuilder 3.2.1 which is incompatible.
Successfully installed Babel-2.9.0 Flask-1.1.2...
...
...
...
sphinx-rtd-theme 0.5.2 has requirement docutils<0.17, but you have docutils 0.17.
ERROR: Service 'airflow-init' failed to build : The command '/bin/bash -o pipefail -e -u -x -c if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true' returned a non-zero code: 1
**What you expected to happen**:
docker-compose build should build without a hitch, and did so before docutils==0.17 was released.
**How to reproduce it**:
Assuming you have docker and docker-compose installed:
1. wget https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml
2. git clone -b v2-0-stable https://github.com/apache/airflow.git
3. Edit docker-compose.yml and make sure the airflow-init target looks like this:
```
airflow-init:
build:
context: ./airflow
args:
PYTHON_BASE_IMAGE: "python:3.8-slim-buster"
PYTHON_MAJOR_MINOR_VERSION: "3.8"
AIRFLOW_VERSION: "2.0.1"
AIRFLOW_EXTRAS: "celery,doc,docker,postgres,redis,amazon,elasticsearch,ftp,grpc,hashicorp,http,jira,kubernetes,openfaas,opsgenie,oracle,pagerduty,papermill,sftp,slack"
AIRFLOW_SOURCES_FROM: "./airflow"
AIRFLOW_SOURCES_TO: "/opt/airflow"
INSTALL_MYSQL_CLIENT: "false"
ADDITIONAL_RUNTIME_APT_DEPS: "vim"
image: test/airflow:2.0.1
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
```
4. docker-compose build
**Anything else we need to know**:
It seems that since docutils was updated to 0.17 on April 3rd, 2021, the building of a custom airflow image broke.
This breaks every time trying to build the airflow docker image.
The complete output from docker-compose build is attached.
[airflow-build-log.txt](https://github.com/apache/airflow/files/6266167/airflow-build-log.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #15231: New version of docutils breaks Dockerfile for tag 2.0.1 (branch v2-0-stable)
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15231:
URL: https://github.com/apache/airflow/issues/15231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15231: New version of docutils breaks Dockerfile for tag 2.0.1 (branch v2-0-stable)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15231:
URL: https://github.com/apache/airflow/issues/15231#issuecomment-814355673
Can you please add `AIRFLOW_CONSTRAINTS_REFERENCE: constraints-2.0.1` build arg and see if this fixes the problem? If so, I will shortly for 2.0.2 so that you would not have to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15231: New version of docutils breaks Dockerfile for tag 2.0.1 (branch v2-0-stable)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15231:
URL: https://github.com/apache/airflow/issues/15231#issuecomment-814366752
I just verified that `AIRFLOW_CONSTRAINTS_REFERENCE: constraints-2.0.1` fixes it.
It's already fixed in 2.0.2 so when we release 2.0.2, you will be able to remove it.
Closing this one as we have both - mitigation and permanent solution in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jackemuk-es commented on issue #15231: New version of docutils breaks Dockerfile for tag 2.0.1 (branch v2-0-stable)
Posted by GitBox <gi...@apache.org>.
jackemuk-es commented on issue #15231:
URL: https://github.com/apache/airflow/issues/15231#issuecomment-814336262
The issue is with the doc extra being installed, removing it from the AIRFLOW_EXTRAS produces the docker image.
Changing the extras line in the docker-compose file to:
AIRFLOW_EXTRAS: "doc"
produces the same results with docutils.
sphinx-rtd-theme 0.5.2 has requirement docutils<0.17, but you have docutils 0.17.
ERROR: Service 'airflow-init' failed to build : The command '/bin/bash -o pipefail -e -u -x -c if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true' returned a non-zero code: 1
gjackemuk@iad-devops-airflow-dev (10.0.2.6, devops): ~/_github/git-airflow-bug$ vi docker-compose.yaml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15231: New version of docutils breaks Dockerfile for tag 2.0.1 (branch v2-0-stable)
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15231:
URL: https://github.com/apache/airflow/issues/15231#issuecomment-814261739
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jackemuk-es commented on issue #15231: New version of docutils breaks Dockerfile for tag 2.0.1 (branch v2-0-stable)
Posted by GitBox <gi...@apache.org>.
jackemuk-es commented on issue #15231:
URL: https://github.com/apache/airflow/issues/15231#issuecomment-814389960
> I just verified that adding `AIRFLOW_CONSTRAINTS_REFERENCE: constraints-2.0.1` fixes it.
>
> It's already fixed in 2.0.2 so when we release 2.0.2, you will be able to remove it.
>
> Closing this one as we have both - mitigation and permanent solution in place.
Thanks! I did verify that does indeed work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15231: New version of docutils breaks Dockerfile for tag 2.0.1 (branch v2-0-stable)
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15231:
URL: https://github.com/apache/airflow/issues/15231#issuecomment-814366752
I just verified that adding `AIRFLOW_CONSTRAINTS_REFERENCE: constraints-2.0.1` fixes it.
It's already fixed in 2.0.2 so when we release 2.0.2, you will be able to remove it.
Closing this one as we have both - mitigation and permanent solution in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org