You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kalluripradeep (via GitHub)" <gi...@apache.org> on 2026/03/08 11:25:31 UTC
[PR] fix: ensure Celery tasks are registered at worker startup (main) [airflow]
kalluripradeep opened a new pull request, #63110:
URL: https://github.com/apache/airflow/pull/63110
Follow-up to #63050
When the Celery worker starts, it imports `app` from `celery_executor.py` but `celery_executor_utils.py` was never imported at module level. This means `execute_workload` (and `execute_command` for Airflow 2.x) are never registered as Celery tasks when the worker boots, causing:
KeyError: 'execute_command'
KeyError: 'execute_workload'
Received unregistered task of type '...'. The message has been ignored and discarded.
Fix is a single module-level import in `celery_executor.py` so `celery_executor_utils` is loaded when the worker imports `app` at startup, ensuring all tasks are registered immediately.
Also adds a regression test to verify task registration on import.
Fixes #63043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: ensure Celery tasks are registered at worker startup (main) [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #63110:
URL: https://github.com/apache/airflow/pull/63110#discussion_r2902361887
##########
providers/celery/tests/unit/celery/executors/test_celery_executor.py:
##########
@@ -687,3 +687,25 @@ def test_task_routing_through_team_specific_app(self, mock_send_tasks, monkeypat
# Critical: task belongs to team A's app, not module-level app
assert task_from_call.app is team_a_executor.celery_app
assert task_from_call.name == "execute_command"
+
+
+def test_celery_tasks_registered_on_import():
Review Comment:
This test function is to be removed when provider min airflow version is >= airflow 3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: ensure Celery tasks are registered at worker startup (main) [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #63110:
URL: https://github.com/apache/airflow/pull/63110#discussion_r2902361887
##########
providers/celery/tests/unit/celery/executors/test_celery_executor.py:
##########
@@ -687,3 +687,25 @@ def test_task_routing_through_team_specific_app(self, mock_send_tasks, monkeypat
# Critical: task belongs to team A's app, not module-level app
assert task_from_call.app is team_a_executor.celery_app
assert task_from_call.name == "execute_command"
+
+
+def test_celery_tasks_registered_on_import():
Review Comment:
Is this test function to be removed when min airflow version is >= airflow 3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: ensure Celery tasks are registered at worker startup (main) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #63110:
URL: https://github.com/apache/airflow/pull/63110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: ensure Celery tasks are registered at worker startup (main) [airflow]
Posted by "kalluripradeep (via GitHub)" <gi...@apache.org>.
kalluripradeep commented on code in PR #63110:
URL: https://github.com/apache/airflow/pull/63110#discussion_r2902455354
##########
providers/celery/tests/unit/celery/executors/test_celery_executor.py:
##########
@@ -687,3 +687,25 @@ def test_task_routing_through_team_specific_app(self, mock_send_tasks, monkeypat
# Critical: task belongs to team A's app, not module-level app
assert task_from_call.app is team_a_executor.celery_app
assert task_from_call.name == "execute_command"
+
+
+def test_celery_tasks_registered_on_import():
Review Comment:
Yes exactly — `execute_command` is Airflow 2.x only. I've added a `# TODO: remove this block when min supported Airflow version is >= 3.0` comment on that assertion to make the cleanup intent explicit. Thanks for flagging!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: ensure Celery tasks are registered at worker startup (main) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63110:
URL: https://github.com/apache/airflow/pull/63110#issuecomment-4024118085
Nice ! Thanks @kalluripradeep !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org