You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vghar-bh (via GitHub)" <gi...@apache.org> on 2023/07/10 09:30:38 UTC

[GitHub] [airflow] vghar-bh opened a new issue, #32484: Celery worker enters a catatonic state after redis restart

vghar-bh opened a new issue, #32484:
URL: https://github.com/apache/airflow/issues/32484

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   Worker seems to be stuck in a catatonic state where queued tasks instance messages are not consumed from redis.
   
   Redis did restart while the worker remained as is. The worker did output logs that indicated a loss in connection but was able to reconnect after redis came back online.
   
   **worker logs:** 
   
   [2023-07-09 01:07:58,950: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 336, in start
       blueprint.start(self)
     File "/home/airflow/.local/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
       step.start(parent)
     File "/home/airflow/.local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 726, in start
       c.loop(*c.loop_args())
     File "/home/airflow/.local/lib/python3.8/site-packages/celery/worker/loops.py", line 97, in asynloop
       next(loop)
     File "/home/airflow/.local/lib/python3.8/site-packages/kombu/asynchronous/hub.py", line 373, in create_loop
       cb(*cbargs)
     File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/redis.py", line 1336, in on_readable
       self.cycle.on_readable(fileno)
     File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/redis.py", line 566, in on_readable
       chan.handlers[type]()
     File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/redis.py", line 910, in _receive
       ret.append(self._receive_one(c))
     File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/redis.py", line 920, in _receive_one
       response = c.parse_response()
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1542, in parse_response
       response = self._execute(conn, try_read)
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1518, in _execute
       return conn.retry.call_with_retry(
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/retry.py", line 49, in call_with_retry
       fail(error)
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1520, in <lambda>
       lambda error: self._disconnect_raise_connect(conn, error),
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1507, in _disconnect_raise_connect
       raise error
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/retry.py", line 46, in call_with_retry
       return do()
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1519, in <lambda>
       lambda: command(*args, **kwargs),
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1540, in try_read
       return conn.read_response(disconnect_on_error=False)
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/connection.py", line 874, in read_response
       response = self._parser.read_response(disable_decoding=disable_decoding)
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/connection.py", line 347, in read_response
       result = self._read_response(disable_decoding=disable_decoding)
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/connection.py", line 357, in _read_response
       raw = self._buffer.readline()
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/connection.py", line 260, in readline
       self._read_from_socket()
     File "/home/airflow/.local/lib/python3.8/site-packages/redis/connection.py", line 213, in _read_from_socket
       raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
   redis.exceptions.ConnectionError: Connection closed by server.
   [2023-07-09 01:07:58,958: WARNING/MainProcess] /home/airflow/.local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:386: CPendingDeprecationWarning:
   In Celery 5.1 we introduced an optional breaking change which
   on connection loss cancels all currently executed tasks with late acknowledgement enabled.
   These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered
   back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss
   setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.
   
     warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)
   
   [2023-07-09 01:07:58,978: WARNING/MainProcess] /home/airflow/.local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
   whether broker connection retries are made during startup in Celery 6.0 and above.
   If you wish to retain the existing behavior for retrying connections on startup,
   you should set broker_connection_retry_on_startup to True.
     warnings.warn(
   
   [2023-07-09 01:07:58,985: ERROR/MainProcess] consumer: Cannot connect to redis://:**@test-airflow-redis-master.test-idp.svc.cluster.local:6379/1: Error 111 connecting to test-airflow-redis-master.test-idp.svc.cluster.local:6379. Connection refused..
   Trying again in 2.00 seconds... (1/100)
   
   [2023-07-09 01:14:09,530: ERROR/MainProcess] consumer: Cannot connect to redis://:**@test-airflow-redis-master.test-idp.svc.cluster.local:6379/1: Error 110 connecting to test-airflow-redis-master.test-idp.svc.cluster.local:6379. Connection timed out..
   Trying again in 32.00 seconds... (16/100)
   
   [2023-07-09 01:14:41,652: INFO/MainProcess] Connected to redis://:**@test-airflow-redis-master.test-idp.svc.cluster.local:6379/1
   [2023-07-09 01:14:41,653: WARNING/MainProcess] /home/airflow/.local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
   whether broker connection retries are made during startup in Celery 6.0 and above.
   If you wish to retain the existing behavior for retrying connections on startup,
   you should set broker_connection_retry_on_startup to True.
     warnings.warn(
   
   [2023-07-09 01:14:41,666: INFO/MainProcess] mingle: searching for neighbors
   [2023-07-09 01:14:42,691: INFO/MainProcess] mingle: all alone
   
   
   
   
   
   
   ### What you think should happen instead
   
   After redis comes back online and the worker connected again, the worker should consume the messages and execute queued task instances.
   
   ### How to reproduce
   
   - Delete the existing redis pod and the worker should be unable to connect to redis
   - Redis restarts and the worker connects as expected
   - Worker does not consume new messages (queued task instances)
   
   ### Operating System
   
   N/A
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon	5.1.0
   apache-airflow-providers-apache-livy 2.2.3	
   apache-airflow-providers-celery	3.0.0
   apache-airflow-providers-cncf-kubernetes 4.4.0
   apache-airflow-providers-common-sql	1.2.0
   apache-airflow-providers-docker	3.2.0
   apache-airflow-providers-elasticsearch	4.2.1
   apache-airflow-providers-ftp	3.1.0
   apache-airflow-providers-google	8.3.0
   apache-airflow-providers-grpc	3.0.0
   apache-airflow-providers-hashicorp	3.1.0
   apache-airflow-providers-http	4.0.0
   apache-airflow-providers-imap	3.0.0
   apache-airflow-providers-microsoft-azure 4.3.0
   apache-airflow-providers-mysql	3.2.1
   apache-airflow-providers-odbc	3.1.2
   apache-airflow-providers-postgres	5.2.2
   apache-airflow-providers-redis	3.0.0
   apache-airflow-providers-sendgrid	3.0.0
   apache-airflow-providers-sftp	4.1.0
   apache-airflow-providers-slack	5.1.0
   apache-airflow-providers-sqlite	3.2.1
   apache-airflow-providers-ssh 3.2.0
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   Airflow version 2.4.1
   
   currently working with helm charts from Airflow Helm Chart (User Community) 
   https://artifacthub.io/packages/helm/airflow-helm/airflow/8.6.1
   
   using  Argocd as GitOps CD tool for Kubernetes.
   
   
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] github-actions[bot] commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1712667768

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] github-actions[bot] closed issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #32484: Celery worker enters a catatonic state after redis restart  and the tasks are getting queued
URL: https://github.com/apache/airflow/issues/32484


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #32484: Celery worker enters a catatonic state after redis restart

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1628584439

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] hussein-awala commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1629676812

   It appears to be an issue with Celery on Redis: https://github.com/celery/celery/issues/8030
   They mentioned that downgrading to celery==5.2.2 could temporary fix the problem, but unfortunately this version is not supported by the Airflow celery provider.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] VladimirYushkevich commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued

Posted by "VladimirYushkevich (via GitHub)" <gi...@apache.org>.
VladimirYushkevich commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1660972279

   We have the same lost connection stacktrace in our `worker` pods (`celery` executor), airflow `2.6.2`. However  setting explicitly celery==5.2.2 didn't help. After triggering task it stuck in queued state on UI and there are no messages about it on worker.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] github-actions[bot] commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1703331910

   This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] hussein-awala commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1629696219

   There is no breaking changes between 5.2.0 and 5.2.3, and bumping the min celery version to 5.2.3 is not to logical for me: https://github.com/apache/airflow/pull/19703
   IMO using celery 5.2.2 should be safe, and it can help you to avoid this issue, but you should force installing it after installing Airflow, you can try:
   ```bash
   pip install --no-deps celery==5.2.2
   ```
   @potiuk I suggest changing the min celery version to 5.2.0 in Airflow 2.6.4, WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1630896889

   Actually - we already have celery>=5.2.3 in celery provider but this is optional. I think for 2.7.0 (hope) when I manage to implement moving celery + kubernetes executors to providers, we might be able to fix it better - for now I think we can leave it as it is (based on constraints)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] hussein-awala commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1631555887

   > I think for 2.7.0 (hope) when I manage to implement moving celery + kubernetes executors to providers, we might be able to fix it better 
   
   I hope so too!
   
   Waiting a response from @vghar-bh 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1630889586

   Sure. Good idea.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org