You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/20 18:32:16 UTC
[GitHub] [airflow] DarthUdp opened a new issue #20996: Celery worker having issues with a seemingly code error
DarthUdp opened a new issue #20996:
URL: https://github.com/apache/airflow/issues/20996
### Apache Airflow version
2.2.3 (latest released)
### What happened
When we try to run dags we get the error described bellow, I'm at a loss for further diagnosis of this issue and thus decided to open this issue, I'm happy to provide any extra details or even help with a fix.
Regards
### What you expected to happen
When we try to run any dag with the celery executor we get the following error (includes the built-in example dags):
```
[2022-01-20 18:22:09,694: ERROR/ForkPoolWorker-16] Failed to execute task return_code() got an unexpected keyword argument 'timeout'.
Traceback (most recent call last):
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 121, in _execute_in_fork
args.func(args)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 298, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 105, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 163, in _run_task_by_local_task_job
run_job.run()
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 245, in run
self._execute()
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 129, in _execute
return_code = self.task_runner.return_code(timeout=max_wait_time)
TypeError: return_code() got an unexpected keyword argument 'timeout'
[2022-01-20 18:22:09,694: ERROR/ForkPoolWorker-15] Failed to execute task return_code() got an unexpected keyword argument 'timeout'.
Traceback (most recent call last):
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 121, in _execute_in_fork
args.func(args)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 298, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 105, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 163, in _run_task_by_local_task_job
run_job.run()
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 245, in run
self._execute()
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 129, in _execute
return_code = self.task_runner.return_code(timeout=max_wait_time)
TypeError: return_code() got an unexpected keyword argument 'timeout'
[2022-01-20 18:22:09,784: ERROR/ForkPoolWorker-15] Task airflow.executors.celery_executor.execute_command[5f812025-28ff-4805-b876-680175d0e51c] raised unexpected: AirflowException('Celery command failed on host: airflow-act.internal.cloudapp.net')
Traceback (most recent call last):
File "/opt/airflow/.env/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/airflow/.env/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 90, in execute_command
_execute_in_fork(command_to_exec, celery_task_id)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 101, in _execute_in_fork
raise AirflowException('Celery command failed on host: ' + get_hostname())
airflow.exceptions.AirflowException: Celery command failed on host: airflow-act.internal.cloudapp.net
[2022-01-20 18:22:09,784: ERROR/ForkPoolWorker-16] Task airflow.executors.celery_executor.execute_command[2911ab23-945b-4526-9971-aadf20a04536] raised unexpected: AirflowException('Celery command failed on host: airflow-act.internal.cloudapp.net')
Traceback (most recent call last):
File "/opt/airflow/.env/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/airflow/.env/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 90, in execute_command
_execute_in_fork(command_to_exec, celery_task_id)
File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 101, in _execute_in_fork
raise AirflowException('Celery command failed on host: ' + get_hostname())
airflow.exceptions.AirflowException: Celery command failed on host: airflow-act.internal.cloudapp.net
```
### How to reproduce
We have installed airflow as per the documentation with the following pip command.
### Operating System
Ubuntu 20.04.3 LTS (Focal Fossa)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-jdbc==2.0.1
apache-airflow-providers-postgres==2.4.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sqlite==2.0.1
```
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
Every time we try to run a DAG.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #20996: Celery worker having issues with a seemingly code error
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20996:
URL: https://github.com/apache/airflow/issues/20996#issuecomment-1017801023
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20996: Celery worker having issues with a seemingly code error
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20996:
URL: https://github.com/apache/airflow/issues/20996#issuecomment-1019563095
I believe your installation has gone somewhat wrong - almost looks like you have mulltple versions of airflow installed at the same time. I suggest you reinstall you Airlfow installation from scratch (alsol you have not really posted how you installed airflow). Converting it into discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org