You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/09 10:17:09 UTC
[GitHub] [airflow] Bowrna opened a new issue #21454: Breeze build-image with dry-run-docker flag command
Bowrna opened a new issue #21454:
URL: https://github.com/apache/airflow/issues/21454
### Apache Airflow version
main (development)
### What happened
I tried to execute the `./breeze build-image --dry-run-docker`
```
Good version of docker 20.10.10.
Dry run mode
Build image
docker buildx build --builder default --progress=tty --pull --build-arg PYTHON_BASE_IMAGE=python:3.7-slim-buster --build-arg AIRFLOW_VERSION=2.3.0.dev0 --build-arg AIRFLOW_BRANCH=main --build-arg AIRFLOW_EXTRAS=devel_ci --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES=true --build-arg ADDITIONAL_AIRFLOW_EXTRAS= --build-arg ADDITIONAL_PYTHON_DEPS= --build-arg ADDITIONAL_DEV_APT_COMMAND= --build-arg ADDITIONAL_DEV_APT_DEPS= --build-arg ADDITIONAL_DEV_APT_ENV= --build-arg ADDITIONAL_RUNTIME_APT_COMMAND= --build-arg ADDITIONAL_RUNTIME_APT_DEPS= --build-arg ADDITIONAL_RUNTIME_APT_ENV= --build-arg UPGRADE_TO_NEWER_DEPENDENCIES=false --build-arg CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow --build-arg AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main --build-arg AIRFLOW_CONSTRAINTS=constraints-source-providers --build-arg AIRFLOW_IMAGE_REPOSITORY=https://github.com/apache/airflow --build-arg AIRFLOW_IMAGE_DATE_CREATED=2022-02-09T09:24:11Z --build-arg BUILD_ID=0 --build-arg COMMIT_SHA=ebf76087147
0446fb5e0cd58edf8fec32547e65e --cache-from=ghcr.io/apache/airflow/main/ci/python3.7:cache -t ghcr.io/apache/airflow/main/ci/python3.7 --target main . -f Dockerfile.ci
CI image.
Branch name: main
Docker image: ghcr.io/apache/airflow/main/ci/python3.7
Airflow source version: 2.3.0.dev0
Python version: 3.7
Backend: sqlite
```
I tried to execute the docker command alone by extracting it out. But it failed to execute in the cache apt-get command while the breeze command worked properly
```
[+] Building 26.8s (8/38)
=> [internal] load build definition from Dockerfile.ci 0.0s
=> => transferring dockerfile: 36B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> [internal] load metadata for docker.io/library/python:3.7-slim-buster 1.5s
=> importing cache manifest from ghcr.io/apache/airflow/main/ci/python3.7:cache 1.6s
=> [internal] load build context 0.6s
=> => transferring context: 417.24kB 0.6s
=> [ 1/33] FROM docker.io/library/python:3.7-slim-buster@sha256:66fbe736b614879d760c5569a775f2d8e6c3d60f9f33c6d39b1048a54f2d4cc0 0.0s
=> CACHED [ 2/33] RUN echo "Base image version: python:3.7-slim-buster" 0.0s
=> ERROR [ 3/33] RUN apt-get update && apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 && apt-get install -y --no-install-recommends curl gnupg2 && mkdir -pv /u 23.6s
------
> [ 3/33] RUN apt-get update && apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 && apt-get install -y --no-install-recommends curl gnupg2 && mkdir -pv /usr/share/man/man1 && mkdir -pv /usr/share/man/man7 && export && bash -o pipefail -o errexit -o nounset -o nolog -c " curl --silent --fail --location https://deb.nodesource.com/setup_14.x | bash - && curl --silent --fail https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - >/dev/null 2>&1 && echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list" && bash -o pipefail -o errexit -o nounset -o nolog -c "" && apt-get update && apt-get install -y --no-install-recommends apt-utils build-essential dirmngr dumb-init freetds-bin freetds-dev git graphviz gosu libffi-dev libldap2-dev libkrb5-dev
libpq-dev libsasl2-2 libsasl2-dev libsasl2-modules libssl-dev libenchant-dev locales netcat nodejs rsync sasl2-bin sudo unixodbc unixodbc-dev yarn && apt-get autoremove -yqq --purge && apt-get clean && rm -rf /var/lib/apt/lists/*:
#7 0.603 Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
#7 0.740 Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
#7 0.878 Get:3 http://security.debian.org/debian-security buster/updates/main arm64 Packages [308 kB]
#7 0.992 Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
#7 1.031 Get:5 http://deb.debian.org/debian buster/main arm64 Packages [7735 kB]
#7 2.133 Get:6 http://deb.debian.org/debian buster-updates/main arm64 Packages [8784 B]
```
### What you expected to happen
I expected the docker command to run successfully like breeze build-image command
### How to reproduce
By executing the below docker command in AIRFLOW_HOME dir
```
docker buildx build --builder default --progress=tty --pull --build-arg PYTHON_BASE_IMAGE=python:3.7-slim-buster --build-arg AIRFLOW_VERSION=2.3.0.dev0 --build-arg AIRFLOW_BRANCH=main --build-arg AIRFLOW_EXTRAS=devel_ci --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES=true --build-arg ADDITIONAL_AIRFLOW_EXTRAS= --build-arg ADDITIONAL_PYTHON_DEPS= --build-arg ADDITIONAL_DEV_APT_COMMAND= --build-arg ADDITIONAL_DEV_APT_DEPS= --build-arg ADDITIONAL_DEV_APT_ENV= --build-arg ADDITIONAL_RUNTIME_APT_COMMAND= --build-arg ADDITIONAL_RUNTIME_APT_DEPS= --build-arg ADDITIONAL_RUNTIME_APT_ENV= --build-arg UPGRADE_TO_NEWER_DEPENDENCIES=false --build-arg CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow --build-arg AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main --build-arg AIRFLOW_CONSTRAINTS=constraints-source-providers --build-arg AIRFLOW_IMAGE_REPOSITORY=https://github.com/apache/airflow --build-arg AIRFLOW_IMAGE_DATE_CREATED=2022-02-09T09:24:11Z --build-arg BUILD_ID=0 --build-arg COMMIT_SHA=ebf76087147
0446fb5e0cd58edf8fec32547e65e --cache-from=ghcr.io/apache/airflow/main/ci/python3.7:cache -t ghcr.io/apache/airflow/main/ci/python3.7 --target main . -f Dockerfile.ci
```
### Operating System
macOS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21454: Breeze build-image with dry-run-docker flag command
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21454:
URL: https://github.com/apache/airflow/issues/21454#issuecomment-1042979916
```
[jarek:~/code/airflow] [breeze] remove-wrapper-script-from-static-checks+ 7s ± ./breeze build-image --dry-run-docker
Good version of docker 20.10.7.
Dry run mode
Build image
docker buildx build --builder default --progress=tty --pull --build-arg PYTHON_BASE_IMAGE=python:3.7-slim-buster --build-arg AIRFLOW_VERSION=2.3.0.dev0 --build-arg AIRFLOW_BRANCH=main --build-arg AIRFLOW_EXTRAS=devel_ci --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES=true --build-arg ADDITIONAL_AIRFLOW_EXTRAS= --build-arg ADDITIONAL_PYTHON_DEPS= --build-arg ADDITIONAL_DEV_APT_COMMAND= --build-arg ADDITIONAL_DEV_APT_DEPS= --build-arg ADDITIONAL_DEV_APT_ENV= --build-arg ADDITIONAL_RUNTIME_APT_COMMAND= --build-arg ADDITIONAL_RUNTIME_APT_DEPS= --build-arg ADDITIONAL_RUNTIME_APT_ENV= --build-arg UPGRADE_TO_NEWER_DEPENDENCIES=false --build-arg CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow --build-arg AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main --build-arg AIRFLOW_CONSTRAINTS=constraints-source-providers --build-arg AIRFLOW_IMAGE_REPOSITORY=https://github.com/apache/airflow --build-arg AIRFLOW_IMAGE_DATE_CREATED=2022-02-17T13:57:46Z --build-arg BUILD_ID=0 --build-arg COMMIT_SHA=5d1857dae1d
3a6d3f712be2043d2aeb99d8c7b12 --cache-from=ghcr.io/apache/airflow/main/ci/python3.7:cache -t ghcr.io/apache/airflow/main/ci/python3.7 --target main . -f Dockerfile.ci
CI image.
Branch name: main
Docker image: ghcr.io/apache/airflow/main/ci/python3.7
Airflow source version: 2.3.0.dev0
Python version: 3.7
Backend: mssql
[jarek:~/code/airflow] [breeze] remove-wrapper-script-from-static-checks+ ± docker buildx build --builder default --progress=tty --pull --build-arg PYTHON_BASE_IMAGE=python:3.7-slim-buster --build-arg AIRFLOW_VERSION=2.3.0.dev0 --build-arg AIRFLOW_BRANCH=main --build-arg AIRFLOW_EXTRAS=devel_ci --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES=true --build-arg ADDITIONAL_AIRFLOW_EXTRAS= --build-arg ADDITIONAL_PYTHON_DEPS= --build-arg ADDITIONAL_DEV_APT_COMMAND= --build-arg ADDITIONAL_DEV_APT_DEPS= --build-arg ADDITIONAL_DEV_APT_ENV= --build-arg ADDITIONAL_RUNTIME_APT_COMMAND= --build-arg ADDITIONAL_RUNTIME_APT_DEPS= --build-arg ADDITIONAL_RUNTIME_APT_ENV= --build-arg UPGRADE_TO_NEWER_DEPENDENCIES=false --build-arg CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow --build-arg AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main --build-arg AIRFLOW_CONSTRAINTS=constraints-source-providers --build-arg AIRFLOW_IMAGE_REPOSITORY=https://github.com/apache/airflow --build-arg AIRFLOW_IMAGE_DATE_CREATED=2
022-02-17T13:57:46Z --build-arg BUILD_ID=0 --build-arg COMMIT_SHA=5d1857dae1d3a6d3f712be2043d2aeb99d8c7b12 --cache-from=ghcr.io/apache/airflow/main/ci/python3.7:cache -t ghcr.io/apache/airflow/main/ci/python3.7 --target main . -f Dockerfile.ci
[+] Building 146.9s (43/43) FINISHED
=> [internal] load build definition from Dockerfile.ci 0.9s
=> => transferring dockerfile: 36B 0.8s
=> [internal] load .dockerignore 0.9s
=> => transferring context: 35B 0.8s
=> [internal] load metadata for docker.io/library/python:3.7-slim-buster 1.9s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> importing cache manifest from ghcr.io/apache/airflow/main/ci/python3.7:cache 1.5s
=> [internal] load build context 1.9s
=> => transferring context: 1.83MB 1.9s
=> [auth] apache/airflow/main/ci/python3.7:pull token for ghcr.io 0.0s
=> [ 1/35] FROM docker.io/library/python:3.7-slim-buster@sha256:66fbe736b614879d760c5569a775f2d8e6c3d60f9f33c6d39b1048a54f2d4cc0 0.0s
=> => resolve docker.io/library/python:3.7-slim-buster@sha256:66fbe736b614879d760c5569a775f2d8e6c3d60f9f33c6d39b1048a54f2d4cc0 0.0s
=> CACHED [ 2/35] RUN echo "Base image version: python:3.7-slim-buster" 0.0s
=> CACHED [ 3/35] RUN apt-get update && apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 && apt-get install -y --no-install-recommends curl gnupg2 && mkdir -pv /usr/share/man/man1 && mkdir -pv /usr/share/man/man7 && export & 0.0s
=> CACHED [ 4/35] COPY scripts/docker/install_mysql.sh scripts/docker/install_mssql.sh /scripts/docker/ 0.0s
=> CACHED [ 5/35] RUN bash /scripts/docker/install_mysql.sh prod && bash /scripts/docker/install_mysql.sh dev && bash /scripts/docker/install_mssql.sh && adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password --quiet " 0.0s
=> CACHED [ 6/35] RUN SYSTEM=$(uname -s | tr '[:upper:]' '[:lower:]') && HELM_URL="https://get.helm.sh/helm-v3.6.3-${SYSTEM}-amd64.tar.gz" && curl --silent --location "${HELM_URL}" | tar -xz -O "${SYSTEM}"-amd64/helm > /usr/local/bin/helm && chmod +x /usr/loc 0.0s
=> CACHED [ 7/35] RUN mkdir -pv /usr/share/man/man1 && mkdir -pv /usr/share/man/man7 && export && export && bash -o pipefail -o errexit -o nounset -o nolog -c "" && bash -o pipefail -o errexit -o nounset -o nolog -c "" && apt-get update && a 0.0s
=> CACHED [ 8/35] WORKDIR /opt/airflow 0.0s
=> CACHED [ 9/35] RUN mkdir -pv /root/airflow && mkdir -pv /root/airflow/dags && mkdir -pv /root/airflow/logs 0.0s
=> CACHED [10/35] RUN echo "Airflow version: 2.3.0.dev0" 0.0s
=> CACHED [11/35] COPY scripts/docker/install_pip_version.sh scripts/docker/install_airflow_dependencies_from_branch_tip.sh scripts/docker/common.sh /scripts/docker/ 0.0s
=> CACHED [12/35] RUN echo -e "\n\e[32mThe 'Running pip as the root user' warnings below are not valid but we can't disable them :(\e[0m\n"; echo -e "\n\e[34mSee https://github.com/pypa/pip/issues/10556 for details.\e[0m\n" ; bash /scripts/docker/install_pip_vers 0.0s
=> CACHED [13/35] COPY scripts/docker/install_pipx_tools.sh /scripts/docker/ 0.0s
=> CACHED [14/35] RUN bash /scripts/docker/install_pipx_tools.sh 0.0s
=> CACHED [15/35] COPY airflow/www/package.json airflow/www/yarn.lock /opt/airflow/airflow/www/ 0.0s
=> CACHED [16/35] COPY scripts/docker/prepare_node_modules.sh /scripts/docker/ 0.0s
=> CACHED [17/35] RUN bash /scripts/docker/prepare_node_modules.sh 0.0s
=> CACHED [18/35] COPY airflow/www/webpack.config.js /opt/airflow/airflow/www/ 0.0s
=> CACHED [19/35] COPY airflow/www/static /opt/airflow/airflow/www/static/ 0.0s
=> CACHED [20/35] COPY scripts/docker/compile_www_assets.sh /scripts/docker/ 0.0s
=> CACHED [21/35] RUN REMOVE_ARTIFACTS="false" BUILD_TYPE="build" bash /scripts/docker/compile_www_assets.sh 0.0s
=> CACHED [22/35] COPY setup.py /opt/airflow/setup.py 0.0s
=> [23/35] COPY setup.cfg /opt/airflow/setup.cfg 0.0s
=> [24/35] COPY airflow/__init__.py /opt/airflow/airflow/__init__.py 0.0s
=> [25/35] COPY scripts/docker/install_airflow.sh /scripts/docker/ 0.0s
=> [26/35] RUN if [[ true == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi 134.3s
=> [27/35] COPY scripts/in_container/entrypoint_ci.sh /entrypoint 0.0s
=> [28/35] RUN chmod a+x /entrypoint 0.4s
=> [29/35] COPY scripts/docker/install_pip_version.sh scripts/docker/install_additional_dependencies.sh /scripts/docker/ 0.0s
=> [30/35] RUN bash /scripts/docker/install_pip_version.sh; if [[ -n "" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi 3.7s
=> [31/35] RUN if command -v airflow; then register-python-argcomplete airflow >> ~/.bashrc ; fi 0.5s
=> [32/35] RUN echo "source /etc/bash_completion" >> ~/.bashrc 0.4s
=> [33/35] COPY . /opt/airflow/ 0.4s
=> [34/35] WORKDIR /opt/airflow 0.0s
=> [35/35] RUN ln -sf /usr/bin/dumb-init /usr/local/bin/dumb-init 0.3s
=> exporting to image 0.6s
=> => exporting layers 0.6s
=> => writing image sha256:d9200196fc6c668e98ca889643197d3c5d466b708d02a11b632fc9d24e5bf100 0.0s
=> => naming to ghcr.io/apache/airflow/main/ci/python3.7
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21454: Breeze build-image with dry-run-docker flag command
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21454:
URL: https://github.com/apache/airflow/issues/21454#issuecomment-1042978668
Hmmm. Very strange . IT works for mre when I run it and even when I copied the docker command...
Can you please rebase and check in the latest main ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on issue #21454: Breeze build-image with dry-run-docker flag command
Posted by GitBox <gi...@apache.org>.
Bowrna commented on issue #21454:
URL: https://github.com/apache/airflow/issues/21454#issuecomment-1042760918
@potiuk This issue still persists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21454: Breeze build-image with dry-run-docker flag command
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #21454:
URL: https://github.com/apache/airflow/issues/21454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21454: Breeze build-image with dry-run-docker flag command
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21454:
URL: https://github.com/apache/airflow/issues/21454#issuecomment-1042724112
This should be fixed now. It was a temporary error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21454: Breeze build-image with dry-run-docker flag command
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21454:
URL: https://github.com/apache/airflow/issues/21454#issuecomment-1052512621
Just for the record - that was a problem with Mac OS handling of `platform` the failure only happened on MacOS because buildkit approach defaults to host platform and not Linux. this is now fixed in the upcoming PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on issue #21454: Breeze build-image with dry-run-docker flag command
Posted by GitBox <gi...@apache.org>.
Bowrna commented on issue #21454:
URL: https://github.com/apache/airflow/issues/21454#issuecomment-1033799501
@potiuk While I could reproduce the issue, I am not sure about the root cause of this issue. Could you tell me how i can start with debugging this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org