You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/05 01:05:38 UTC
[GitHub] [airflow] ExecutionSpeed opened a new issue #16803: DockerOperator not working from containerized Airflow
ExecutionSpeed opened a new issue #16803:
URL: https://github.com/apache/airflow/issues/16803
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 2.1.1 -- Docker Image: `apache/airflow:2.1.1-python3.8`
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): Not running on k8s.
**Environment**:
- **Cloud provider or hardware configuration**: DigitalOcean
- **OS** (e.g. from /etc/os-release): Ubuntu
- **Kernel** (e.g. `uname -a`): `Linux airflow 5.11.0-22-generic #23-Ubuntu SMP Thu Jun 17 00:34:23 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux`
- **Install tools**: Installed via Docker & Docker Compose following instructions from [official docker-compose installation docs](https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#docker-compose-yaml).
- **Others**: N/A
**What happened**:
<!-- (please include exact error messages if you can) -->
**What you expected to happen**:
<!-- What do you think went wrong? -->
**How to reproduce it**:
<!---
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
Running with the `DockerOperator` causes the following error:
```
*** Reading local file: /opt/airflow/logs/docker_eod_us_equities/docker_command_sleep/2021-07-04T23:19:48.431208+00:00/2.log
[2021-07-04 23:49:53,886] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: docker_eod_us_equities.docker_command_sleep 2021-07-04T23:19:48.431208+00:00 [queued]>
[2021-07-04 23:49:53,902] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: docker_eod_us_equities.docker_command_sleep 2021-07-04T23:19:48.431208+00:00 [queued]>
[2021-07-04 23:49:53,903] {taskinstance.py:1087} INFO -
--------------------------------------------------------------------------------
[2021-07-04 23:49:53,903] {taskinstance.py:1088} INFO - Starting attempt 2 of 2
[2021-07-04 23:49:53,903] {taskinstance.py:1089} INFO -
--------------------------------------------------------------------------------
[2021-07-04 23:49:53,911] {taskinstance.py:1107} INFO - Executing <Task(DockerOperator): docker_command_sleep> on 2021-07-04T23:19:48.431208+00:00
[2021-07-04 23:49:53,914] {standard_task_runner.py:52} INFO - Started process 3583 to run task
[2021-07-04 23:49:53,920] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'docker_eod_us_equities', 'docker_command_sleep', '2021-07-04T23:19:48.431208+00:00', '--job-id', '52', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/eod_us_equities.py', '--cfg-path', '/tmp/tmp5iz14yg9', '--error-file', '/tmp/tmpleaymjfa']
[2021-07-04 23:49:53,920] {standard_task_runner.py:77} INFO - Job 52: Subtask docker_command_sleep
[2021-07-04 23:49:54,016] {logging_mixin.py:104} INFO - Running <TaskInstance: docker_eod_us_equities.docker_command_sleep 2021-07-04T23:19:48.431208+00:00 [running]> on host 0dff7922cb76
[2021-07-04 23:49:54,172] {taskinstance.py:1300} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=***
AIRFLOW_CTX_DAG_ID=docker_eod_us_equities
AIRFLOW_CTX_TASK_ID=docker_command_sleep
AIRFLOW_CTX_EXECUTION_DATE=2021-07-04T23:19:48.431208+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-07-04T23:19:48.431208+00:00
[2021-07-04 23:49:54,205] {docker.py:231} INFO - Starting docker container from image alpine
[2021-07-04 23:49:54,216] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/home/airflow/.local/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.30/containers/create?name=docker_command_sleep
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 319, in execute
return self._run_image()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 237, in _run_image
self.container = self.cli.create_container(
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/container.py", line 430, in create_container
return self.create_container_from_config(config, name)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/container.py", line 441, in create_container_from_config
return self._result(res, True)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
self._raise_for_status(response)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.30/containers/create?name=docker_command_sleep: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmpw5gvv6dj")
[2021-07-04 23:49:54,222] {taskinstance.py:1544} INFO - Marking task as FAILED. dag_id=docker_eod_us_equities, task_id=docker_command_sleep, execution_date=20210704T231948, start_date=20210704T234953, end_date=20210704T234954
[2021-07-04 23:49:54,297] {local_task_job.py:151} INFO - Task exited with return code 1
```
I have even tried with Docker API v1.41 (latest) and same issue. I have bound the `/var/run/docker.sock` as a bind mount into the container.
**Docker Compose:**
```yaml
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
# Default: apache/airflow:master-python3.8
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
# Default: 50000
#
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account (if requested).
# Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account (if requested).
# Default: airflow
# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
# Default: ''
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.1-python3.8}
environment: &airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 300 # Just to have a fast load in the front-end. Do not use it in production with those configurations.
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true' # "_run_image of the DockerOperator returns now a python string, not a byte string" Ref: https://github.com/apache/airflow/issues/13487
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- '/var/run/docker.sock:/var/run/docker.sock' # We will pass the Docker Deamon as a volume to allow the webserver containers start docker images. Ref: https://stackoverflow.com/q/51342810/7024760
user: '${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}'
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ['CMD', 'pg_isready', '-U', 'airflow']
interval: 5s
retries: 5
restart: always
redis:
image: redis:latest
ports:
- 6379:6379
healthcheck:
test: ['CMD', 'redis-cli', 'ping']
interval: 5s
timeout: 30s
retries: 50
restart: always
airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- 80:8080
healthcheck:
test: ['CMD', 'curl', '--fail', 'http://localhost:80/health']
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-scheduler:
<<: *airflow-common
command: scheduler
healthcheck:
test:
[
'CMD-SHELL',
'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"',
]
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-worker:
<<: *airflow-common
command: celery worker
healthcheck:
test:
- 'CMD-SHELL'
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-init:
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
flower:
<<: *airflow-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ['CMD', 'curl', '--fail', 'http://localhost:5555/']
interval: 10s
timeout: 10s
retries: 5
restart: always
volumes:
postgres-db-volume:
```
**DAG:**
```python
from datetime import datetime, timedelta
import pendulum
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.providers.docker.operators.docker import DockerOperator
from airflow.operators.dummy_operator import DummyOperator
AMERICA_NEW_YORK_TIMEZONE = pendulum.timezone('US/Eastern')
default_args = {
'owner': 'airflow',
'description': 'Docker testing',
'depend_on_past': False,
'start_date': datetime(2021, 5, 1, tzinfo=AMERICA_NEW_YORK_TIMEZONE),
'retries': 1,
'retry_delay': timedelta(minutes=30),
}
with DAG(
'docker_test',
default_args=default_args,
schedule_interval="15 20 * * *",
catchup=False,
) as dag:
start_dag = DummyOperator(task_id='start_dag')
end_dag = DummyOperator(task_id='end_dag')
t1 = BashOperator(task_id='print_current_date', bash_command='date')
t2 = DockerOperator(
task_id='docker_command_sleep',
image='alpine',
container_name='docker_command_sleep',
api_version='1.30',
auto_remove=True,
command="/bin/sleep 3",
docker_url="unix://var/run/docker.sock",
network_mode="bridge",
do_xcom_push=True,
)
start_dag >> t1
t1 >> t2
t2 >> end_dag
```
**Anything else we need to know**: Problem happens anything `DockerOperator` is being used. Not entirely sure why this happening given that the docker sock is fully permissive (has `777`) and is bind mounted into the container. When I test via *docker-py* client in Python shell under `airflow` user inside the container, it works perfectly fine to run all docker-py operations like listing running containers and such confirming the mounted docker UNIX socket is available and working. However, even with the `docker_url` pointing to the docker socket in the above DAG, I am getting this error thrown in above trace.
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16803: DockerOperator not working from containerized Airflow
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-873708659
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-878470378
Seems it can also be workarounded by downgrading to `1.0.1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ledo01 commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
ledo01 commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-879112539
Hi, I got the same issue when updating from 2.1.0 to 2.1.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-877847953
I looked at this and turned the temp file mounting into optional feature that you might disable: see #16932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ExecutionSpeed edited a comment on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
ExecutionSpeed edited a comment on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-874246604
> Can you also try to mount /tmp folder into the container ? I am not telling it's the final solution yet, but it's likely to help or at least validate a hypothesis i have. It looks like the apis used by the DockerOperator create and use a temporary file to pass something between the client and engine.
@potiuk This ended up working with mounting `/tmp` into the container alongside the `/var/run/docker.sock`.
**Docker Compose:**
```yml
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
# Default: apache/airflow:master-python3.8
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
# Default: 50000
#
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account (if requested).
# Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account (if requested).
# Default: airflow
# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
# Default: ''
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.1-python3.8}
environment: &airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 300 # Just to have a fast load in the front-end. Do not use it in production with those configurations.
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true' # "_run_image of the DockerOperator returns now a python string, not a byte string" Ref: https://github.com/apache/airflow/issues/13487
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- '/var/run/docker.sock:/var/run/docker.sock' # We will pass the Docker Deamon as a volume to allow the webserver containers start docker images. Ref: https://stackoverflow.com/q/51342810/7024760
- '/tmp:/tmp'
user: '${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}'
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ['CMD', 'pg_isready', '-U', 'airflow']
interval: 5s
retries: 5
restart: always
redis:
image: redis:latest
ports:
- 6379:6379
healthcheck:
test: ['CMD', 'redis-cli', 'ping']
interval: 5s
timeout: 30s
retries: 50
restart: always
airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- 80:8080
healthcheck:
test: ['CMD', 'curl', '--fail', 'http://localhost:80/health']
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-scheduler:
<<: *airflow-common
command: scheduler
healthcheck:
test:
[
'CMD-SHELL',
'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"',
]
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-worker:
<<: *airflow-common
command: celery worker
healthcheck:
test:
- 'CMD-SHELL'
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-init:
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
flower:
<<: *airflow-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ['CMD', 'curl', '--fail', 'http://localhost:5555/']
interval: 10s
timeout: 10s
retries: 5
restart: always
volumes:
postgres-db-volume:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-873920645
Can you also try to mount /tmp folder into the container ? I am not telling it's the final solution yet, but it's likely to help or at least validate a hypothesis i have. It looks like the apis used by the DockerOperator create and use a temporary file to pass something between the client and engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-878470378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ExecutionSpeed commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
ExecutionSpeed commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-874246604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-874104406
`/var/run/docker.sock'` is owned by the user that doesn't exist in the container, so we don't have access to it.
As a workaround, you can use a sidecar container with `tecnativa/docker-socket-proxy:0.1.1`. For details, see: https://github.com/apache/airflow/issues/8605#issuecomment-784273838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-874269243
Yeah. Thought so. We will likely have to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-874186882
> /var/run/docker.sock' is owned by the user that doesn't exist in the container, so we don't have access to it.
I think that's not it. The access right are good and confirmed they work with docker command (see the description). The real reason is described in #16808 - as I suspected, DockerOperator creates a folder in "/tmp" and tries to bind-mount it to the docker container it runs - which does not work because the "/tmp" file is created inside airflow container, not in the host. This also prevents to use DockerOperator from running with remote engine (because the host folder cannot be bind-mounted there).
The temporary solution for docker-compose I proposed (mounting /tmp directory) should work in this case, but we have to fix it differently - DockerOperator should create a separate volume in docker and copy all the files there and mount this volume in the Container it runs, rather than mount the "/tmp" host folder. This should solve both problem - running airflow as docker container in local docker-compose and running airflow with remote docker engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ExecutionSpeed edited a comment on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
ExecutionSpeed edited a comment on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-874246604
> Can you also try to mount /tmp folder into the container ? I am not telling it's the final solution yet, but it's likely to help or at least validate a hypothesis i have. It looks like the apis used by the DockerOperator create and use a temporary file to pass something between the client and engine.
@potiuk This ended up working with mounting `/tmp` into the container alongside the `/var/run/docker.sock`.
**Docker Compose:**
```yml
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
# Default: apache/airflow:master-python3.8
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
# Default: 50000
#
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account (if requested).
# Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account (if requested).
# Default: airflow
# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
# Default: ''
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.1-python3.8}
environment: &airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 300 # Just to have a fast load in the front-end. Do not use it in production with those configurations.
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true' # "_run_image of the DockerOperator returns now a python string, not a byte string" Ref: https://github.com/apache/airflow/issues/13487
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- '/var/run/docker.sock:/var/run/docker.sock' # We will pass the Docker Deamon as a volume to allow the webserver containers start docker images. Ref: https://stackoverflow.com/q/51342810/7024760
- '/tmp:/tmp'
user: '${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}'
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ['CMD', 'pg_isready', '-U', 'airflow']
interval: 5s
retries: 5
restart: always
redis:
image: redis:latest
ports:
- 6379:6379
healthcheck:
test: ['CMD', 'redis-cli', 'ping']
interval: 5s
timeout: 30s
retries: 50
restart: always
airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- 80:8080
healthcheck:
test: ['CMD', 'curl', '--fail', 'http://localhost:80/health']
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-scheduler:
<<: *airflow-common
command: scheduler
healthcheck:
test:
[
'CMD-SHELL',
'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"',
]
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-worker:
<<: *airflow-common
command: celery worker
healthcheck:
test:
- 'CMD-SHELL'
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-init:
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
flower:
<<: *airflow-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ['CMD', 'curl', '--fail', 'http://localhost:5555/']
interval: 10s
timeout: 10s
retries: 5
restart: always
volumes:
postgres-db-volume:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-873920645
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ExecutionSpeed commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
ExecutionSpeed commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-874361069
@potiuk Great, thank you for the quick response and pointing me to the temporary workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ExecutionSpeed commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
ExecutionSpeed commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-874246604
> Can you also try to mount /tmp folder into the container ? I am not telling it's the final solution yet, but it's likely to help or at least validate a hypothesis i have. It looks like the apis used by the DockerOperator create and use a temporary file to pass something between the client and engine.
@potiuk This ended up working.
**Docker Compose:**
```yml
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
# Default: apache/airflow:master-python3.8
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
# Default: 50000
#
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account (if requested).
# Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account (if requested).
# Default: airflow
# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
# Default: ''
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.1-python3.8}
environment: &airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 300 # Just to have a fast load in the front-end. Do not use it in production with those configurations.
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true' # "_run_image of the DockerOperator returns now a python string, not a byte string" Ref: https://github.com/apache/airflow/issues/13487
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- '/var/run/docker.sock:/var/run/docker.sock' # We will pass the Docker Deamon as a volume to allow the webserver containers start docker images. Ref: https://stackoverflow.com/q/51342810/7024760
- '/tmp:/tmp'
user: '${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}'
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ['CMD', 'pg_isready', '-U', 'airflow']
interval: 5s
retries: 5
restart: always
redis:
image: redis:latest
ports:
- 6379:6379
healthcheck:
test: ['CMD', 'redis-cli', 'ping']
interval: 5s
timeout: 30s
retries: 50
restart: always
airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- 80:8080
healthcheck:
test: ['CMD', 'curl', '--fail', 'http://localhost:80/health']
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-scheduler:
<<: *airflow-common
command: scheduler
healthcheck:
test:
[
'CMD-SHELL',
'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"',
]
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-worker:
<<: *airflow-common
command: celery worker
healthcheck:
test:
- 'CMD-SHELL'
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-init:
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
flower:
<<: *airflow-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ['CMD', 'curl', '--fail', 'http://localhost:5555/']
interval: 10s
timeout: 10s
retries: 5
restart: always
volumes:
postgres-db-volume:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #16803:
URL: https://github.com/apache/airflow/issues/16803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16803: DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock`
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16803:
URL: https://github.com/apache/airflow/issues/16803#issuecomment-874104406
`/var/run/docker.sock'` is owned by the user that doesn't exist in the container, so we don't have access to it.
As a workaround, you can use a sidecar container with `tecnativa/docker-socket-proxy:0.1.1`. For details, see: https://github.com/apache/airflow/issues/8605#issuecomment-784273838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org