You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/30 16:42:02 UTC
[GitHub] [airflow] Flogue opened a new issue, #26807: Celery worker failed to execute task from subdag
Flogue opened a new issue, #26807:
URL: https://github.com/apache/airflow/issues/26807
### Apache Airflow version
2.4.0
### What happened
Celery worker reports "Failed to execute task Task <task name> not found." when the task is in a subdag.
I ran a dag just calling a subdag running a few different operators (PostgresOperator, EmrAddStepsOperator, EmrStepSensor).
The results are always the same.
The task logs can't be retrieved:
``` *** Log file does not exist: /opt/airflow/logs/dag_id=enrichment_ingest.ontrac_enrichment_ingest_subdag/run_id=scheduled__2022-09-30T15:01:19.197333+00:00/task_id=do_idempotent_sql/attempt=1.log
*** Fetching from: http://:8793/log/dag_id=enrichment_ingest.ontrac_enrichment_ingest_subdag/run_id=scheduled__2022-09-30T15:01:19.197333+00:00/task_id=do_idempotent_sql/attempt=1.log
*** Failed to fetch log file from worker. Request URL is missing an 'http://' or 'https://' protocol.```
### What you think should happen instead
Subdags should still work, I need time to remove them from my infrastructure until 3.0 :smile:
### How to reproduce
Try to run "example_subdag_operator.py" from [airflow](https://github.com/apache/airflow)/[airflow](https://github.com/apache/airflow/tree/main/airflow)/[example_dags](https://github.com/apache/airflow/tree/main/airflow/example_dags)/ but in subdag.py, replace the "EmptyOperator" with "PostgresOperator" for example. (if you use the EmptyOperator, the UI and worker wont complain but the logs will still be unavailable.)
### Operating System
Official Docker container (Debian 11)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==5.0.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.3.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-docker==3.1.0
apache-airflow-providers-elasticsearch==4.2.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-azure==4.2.0
apache-airflow-providers-mysql==3.2.0
apache-airflow-providers-odbc==3.1.1
apache-airflow-providers-postgres==5.2.1
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==4.0.0
apache-airflow-providers-slack==5.1.0
apache-airflow-providers-snowflake==3.2.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.1.0
### Deployment
Docker-Compose
### Deployment details
Using the Official Docker image for 2.4 with the provided docker-compose.yaml file.
### Anything else
Celery logs:
```[2022-09-30 12:34:07,125: INFO/MainProcess] Task airflow.executors.celery_executor.execute_command[90dc25a1-899f-4f2c-8651-249529c82023] received
[2022-09-30 12:34:07,148: INFO/ForkPoolWorker-16] [90dc25a1-899f-4f2c-8651-249529c82023] Executing command in Celery: ['airflow', 'tasks', 'run', 'enrichment_ingest.mysql_enrichment_ingest_subdag', 'submit_step', 'scheduled__2022-09-29T12:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/enrichment_ingest.py']
[2022-09-30 12:34:08,895: ERROR/ForkPoolWorker-16] [90dc25a1-899f-4f2c-8651-249529c82023] Failed to execute task Task submit_step not found.
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 130, in _execute_in_fork
args.func(args)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 52, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 378, in task_run
task = dag.get_task(task_id=args.task_id)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py", line 2244, in get_task
raise TaskNotFound(f"Task {task_id} not found")
airflow.exceptions.TaskNotFound: Task submit_step not found
[2022-09-30 12:34:08,915: ERROR/ForkPoolWorker-16] Task airflow.executors.celery_executor.execute_command[90dc25a1-899f-4f2c-8651-249529c82023] raised unexpected: AirflowException('Celery command failed on host: 2b03ca67350a with celery_task_id 90dc25a1-899f-4f2c-8651-249529c82023')
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 96, in execute_command
_execute_in_fork(command_to_exec, celery_task_id)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 111, in _execute_in_fork
raise AirflowException(msg)
airflow.exceptions.AirflowException: Celery command failed on host: 2b03ca67350a with celery_task_id 90dc25a1-899f-4f2c-8651-249529c82023```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] notatallshaw-gts commented on issue #26807: Celery worker failed to execute task from subdag
Posted by GitBox <gi...@apache.org>.
notatallshaw-gts commented on issue #26807:
URL: https://github.com/apache/airflow/issues/26807#issuecomment-1265701346
No idea about subdags but FYI your log issue should be fixed in Airflow 2.4.1: https://github.com/apache/airflow/pull/26493
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #26807: Celery worker failed to execute task from subdag
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #26807: Celery worker failed to execute task from subdag
URL: https://github.com/apache/airflow/issues/26807
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26807: Celery worker failed to execute task from subdag
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26807:
URL: https://github.com/apache/airflow/issues/26807#issuecomment-1272816103
Closing as fixed in 2.4.1. Please double-check @Flogue by migrating - we can always reopen if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org