You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/20 09:47:49 UTC
[GitHub] [airflow] schustmi opened a new issue, #28487: `airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS
schustmi opened a new issue, #28487:
URL: https://github.com/apache/airflow/issues/28487
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Airflow version 2.4.3 running on MacOS, running a DAG using DockerOperators fails in combination with a local standalone deployment:
```
*** Reading local file: /Users/schustmi/airflow/logs/dag_id=test/run_id=manual__2022-12-20T09:42:32.767926+00:00/task_id=docker_task/attempt=1.log
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: test.docker_task manual__2022-12-20T09:42:32.767926+00:00 [queued]>
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: test.docker_task manual__2022-12-20T09:42:32.767926+00:00 [queued]>
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1383} INFO - Executing <Task(DockerOperator): docker_task> on 2022-12-20 09:42:32.767926+00:00
[2022-12-20, 09:42:35 UTC] {standard_task_runner.py:55} INFO - Started process 98208 to run task
[2022-12-20, 09:42:35 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'test', 'docker_task', 'manual__2022-12-20T09:42:32.767926+00:00', '--job-id', '17', '--raw', '--subdir', 'DAGS_FOLDER/dag.py', '--cfg-path', '/var/folders/45/1tkl8h1d3tvf2q72t8p6bw5r0000gn/T/tmpe6h4cv9f']
[2022-12-20, 09:42:35 UTC] {standard_task_runner.py:83} INFO - Job 17: Subtask o
[2022-12-20, 09:42:35 UTC] {task_command.py:376} INFO - Running <TaskInstance: test.docker_task manual__2022-12-20T09:42:32.767926+00:00 [running]> on host schustmi-mac-work.fritz.box
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1590} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=test
AIRFLOW_CTX_TASK_ID=docker_task
AIRFLOW_CTX_EXECUTION_DATE=2022-12-20T09:42:32.767926+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-12-20T09:42:32.767926+00:00
[2022-12-20, 09:42:35 UTC] {local_task_job.py:159} INFO - Task exited with return code Negsignal.SIGSEGV
[2022-12-20, 09:42:35 UTC] {taskinstance.py:2623} INFO - 0 downstream tasks scheduled from follow-on schedule check
```
### What you think should happen instead
On other operating systems, the same combination of standalone deployment and DockerOperators works.
### How to reproduce
```bash
pip install apache-airflow==2.4.3 apache-airflow-providers-docker==3.3.0
airflow standalone
```
Then run the following DAG:
```python
from airflow import DAG
from airflow.providers.docker.operators.docker import DockerOperator
from datetime import datetime
with DAG(dag_id="test", schedule="@once", start_date=datetime.utcnow()) as dag:
docker_task = DockerOperator(image="alpine:latest", command="echo test", task_id="docker_task")
```
### Operating System
MacOS Ventura 13.0 (22A380)
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql 1.3.1
apache-airflow-providers-docker 3.3.0
apache-airflow-providers-ftp 3.2.0
apache-airflow-providers-http 4.1.0
apache-airflow-providers-imap 3.1.0
apache-airflow-providers-sqlite 3.3.1
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] schustmi commented on issue #28487: `airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS
Posted by GitBox <gi...@apache.org>.
schustmi commented on issue #28487:
URL: https://github.com/apache/airflow/issues/28487#issuecomment-1359145525
The same error happens on non-M1 macs, but the `no_proxy=*` workaround in the PR you linked still worked, thanks!
Is there a general option to run tasks without forking? For example setting `AIRFLOW__CORE__DEFAULT_IMPERSONATION=...` also resolves the issue for me, as `StandardTaskRunner.start()` then runs without forking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28487: `airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28487:
URL: https://github.com/apache/airflow/issues/28487#issuecomment-1359097844
This is problem with MacOS M1 not Airflow, not even Docker. There are various workarounds for this. We've implemented a better message for this case in https://github.com/apache/airflow/pull/27381 but due to the nature of this error, we cannot do more - because it is really problem of some of the libraries that do not work on M1 (ARM) Macbook. There are various workarouds you can apply (see the message in #27381) and you can also look for some other cases (search for SIGSEGV our issues or in SIGESEGV MacOS M1 via google).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28487: `airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28487:
URL: https://github.com/apache/airflow/issues/28487#issuecomment-1359153982
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execute-tasks-new-python-interpreter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28487: `airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28487:
URL: https://github.com/apache/airflow/issues/28487#issuecomment-1359154568
And yes we think that on MacOS we should enable it by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28487: `airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28487:
URL: https://github.com/apache/airflow/issues/28487#issuecomment-1359091841
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28487: `airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28487:
URL: https://github.com/apache/airflow/issues/28487#issuecomment-1384048066
> Could you share a link to the solution? I couldn't find a workaround in the linked PR?
Because there is not one, generally accepted - SIGSEGV **might** be generated by various causes and libraries and if you look through linked issues (and maybe search for just SIGSEGV) and read what worked for various people - maybe you will find a solution that will work for you.
One generall thing you can attempt is to set this config to True: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execute-tasks-new-python-interpreter
This **might** help in a number of similar cases but there is no guarantee - because there might be various reasons and libraries raising SIGSEGV. The linked PR might help you to see how you can try to narrow it down and possibly find a root cause.
That thing is - This is not an Airflow Problem - this is a problem that some libraries are not working on ARM and some of them might require some workarounds - but you have to find those workarounds on your own - first attempt various workarouds people found for their libraries, if it does not not know, finding which library causes it and maybe finding or figuring out a new workaround you might need.
Sorry we can't help more, but other than helping you to investigate by better information and showing how you can investigate it is the best we can do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #28487: `airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #28487: `airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS
URL: https://github.com/apache/airflow/issues/28487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kajen3 commented on issue #28487: `airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS
Posted by GitBox <gi...@apache.org>.
kajen3 commented on issue #28487:
URL: https://github.com/apache/airflow/issues/28487#issuecomment-1383866301
> The same error happens on non-M1 macs, but the `no_proxy=*` workaround in the PR you linked still worked, thanks!
>
> Is there a general option to run tasks without forking? For example setting `AIRFLOW__CORE__DEFAULT_IMPERSONATION=...` also resolves the issue for me, as `StandardTaskRunner.start()` then runs without forking.
Could you share a link to the solution? I couldn't find a workaround in the linked PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org