You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vghar-bh (via GitHub)" <gi...@apache.org> on 2023/07/10 09:30:38 UTC
[GitHub] [airflow] vghar-bh opened a new issue, #32484: Celery worker enters a catatonic state after redis restart
vghar-bh opened a new issue, #32484:
URL: https://github.com/apache/airflow/issues/32484
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Worker seems to be stuck in a catatonic state where queued tasks instance messages are not consumed from redis.
Redis did restart while the worker remained as is. The worker did output logs that indicated a loss in connection but was able to reconnect after redis came back online.
**worker logs:**
[2023-07-09 01:07:58,950: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 336, in start
blueprint.start(self)
File "/home/airflow/.local/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/home/airflow/.local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 726, in start
c.loop(*c.loop_args())
File "/home/airflow/.local/lib/python3.8/site-packages/celery/worker/loops.py", line 97, in asynloop
next(loop)
File "/home/airflow/.local/lib/python3.8/site-packages/kombu/asynchronous/hub.py", line 373, in create_loop
cb(*cbargs)
File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/redis.py", line 1336, in on_readable
self.cycle.on_readable(fileno)
File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/redis.py", line 566, in on_readable
chan.handlers[type]()
File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/redis.py", line 910, in _receive
ret.append(self._receive_one(c))
File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/redis.py", line 920, in _receive_one
response = c.parse_response()
File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1542, in parse_response
response = self._execute(conn, try_read)
File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1518, in _execute
return conn.retry.call_with_retry(
File "/home/airflow/.local/lib/python3.8/site-packages/redis/retry.py", line 49, in call_with_retry
fail(error)
File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1520, in <lambda>
lambda error: self._disconnect_raise_connect(conn, error),
File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1507, in _disconnect_raise_connect
raise error
File "/home/airflow/.local/lib/python3.8/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1519, in <lambda>
lambda: command(*args, **kwargs),
File "/home/airflow/.local/lib/python3.8/site-packages/redis/client.py", line 1540, in try_read
return conn.read_response(disconnect_on_error=False)
File "/home/airflow/.local/lib/python3.8/site-packages/redis/connection.py", line 874, in read_response
response = self._parser.read_response(disable_decoding=disable_decoding)
File "/home/airflow/.local/lib/python3.8/site-packages/redis/connection.py", line 347, in read_response
result = self._read_response(disable_decoding=disable_decoding)
File "/home/airflow/.local/lib/python3.8/site-packages/redis/connection.py", line 357, in _read_response
raw = self._buffer.readline()
File "/home/airflow/.local/lib/python3.8/site-packages/redis/connection.py", line 260, in readline
self._read_from_socket()
File "/home/airflow/.local/lib/python3.8/site-packages/redis/connection.py", line 213, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
[2023-07-09 01:07:58,958: WARNING/MainProcess] /home/airflow/.local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:386: CPendingDeprecationWarning:
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered
back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss
setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.
warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)
[2023-07-09 01:07:58,978: WARNING/MainProcess] /home/airflow/.local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
[2023-07-09 01:07:58,985: ERROR/MainProcess] consumer: Cannot connect to redis://:**@test-airflow-redis-master.test-idp.svc.cluster.local:6379/1: Error 111 connecting to test-airflow-redis-master.test-idp.svc.cluster.local:6379. Connection refused..
Trying again in 2.00 seconds... (1/100)
[2023-07-09 01:14:09,530: ERROR/MainProcess] consumer: Cannot connect to redis://:**@test-airflow-redis-master.test-idp.svc.cluster.local:6379/1: Error 110 connecting to test-airflow-redis-master.test-idp.svc.cluster.local:6379. Connection timed out..
Trying again in 32.00 seconds... (16/100)
[2023-07-09 01:14:41,652: INFO/MainProcess] Connected to redis://:**@test-airflow-redis-master.test-idp.svc.cluster.local:6379/1
[2023-07-09 01:14:41,653: WARNING/MainProcess] /home/airflow/.local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
[2023-07-09 01:14:41,666: INFO/MainProcess] mingle: searching for neighbors
[2023-07-09 01:14:42,691: INFO/MainProcess] mingle: all alone
### What you think should happen instead
After redis comes back online and the worker connected again, the worker should consume the messages and execute queued task instances.
### How to reproduce
- Delete the existing redis pod and the worker should be unable to connect to redis
- Redis restarts and the worker connects as expected
- Worker does not consume new messages (queued task instances)
### Operating System
N/A
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon 5.1.0
apache-airflow-providers-apache-livy 2.2.3
apache-airflow-providers-celery 3.0.0
apache-airflow-providers-cncf-kubernetes 4.4.0
apache-airflow-providers-common-sql 1.2.0
apache-airflow-providers-docker 3.2.0
apache-airflow-providers-elasticsearch 4.2.1
apache-airflow-providers-ftp 3.1.0
apache-airflow-providers-google 8.3.0
apache-airflow-providers-grpc 3.0.0
apache-airflow-providers-hashicorp 3.1.0
apache-airflow-providers-http 4.0.0
apache-airflow-providers-imap 3.0.0
apache-airflow-providers-microsoft-azure 4.3.0
apache-airflow-providers-mysql 3.2.1
apache-airflow-providers-odbc 3.1.2
apache-airflow-providers-postgres 5.2.2
apache-airflow-providers-redis 3.0.0
apache-airflow-providers-sendgrid 3.0.0
apache-airflow-providers-sftp 4.1.0
apache-airflow-providers-slack 5.1.0
apache-airflow-providers-sqlite 3.2.1
apache-airflow-providers-ssh 3.2.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Airflow version 2.4.1
currently working with helm charts from Airflow Helm Chart (User Community)
https://artifacthub.io/packages/helm/airflow-helm/airflow/8.6.1
using Argocd as GitOps CD tool for Kubernetes.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1712667768
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued
URL: https://github.com/apache/airflow/issues/32484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #32484: Celery worker enters a catatonic state after redis restart
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1628584439
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1629676812
It appears to be an issue with Celery on Redis: https://github.com/celery/celery/issues/8030
They mentioned that downgrading to celery==5.2.2 could temporary fix the problem, but unfortunately this version is not supported by the Airflow celery provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VladimirYushkevich commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued
Posted by "VladimirYushkevich (via GitHub)" <gi...@apache.org>.
VladimirYushkevich commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1660972279
We have the same lost connection stacktrace in our `worker` pods (`celery` executor), airflow `2.6.2`. However setting explicitly celery==5.2.2 didn't help. After triggering task it stuck in queued state on UI and there are no messages about it on worker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1703331910
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1629696219
There is no breaking changes between 5.2.0 and 5.2.3, and bumping the min celery version to 5.2.3 is not to logical for me: https://github.com/apache/airflow/pull/19703
IMO using celery 5.2.2 should be safe, and it can help you to avoid this issue, but you should force installing it after installing Airflow, you can try:
```bash
pip install --no-deps celery==5.2.2
```
@potiuk I suggest changing the min celery version to 5.2.0 in Airflow 2.6.4, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1630896889
Actually - we already have celery>=5.2.3 in celery provider but this is optional. I think for 2.7.0 (hope) when I manage to implement moving celery + kubernetes executors to providers, we might be able to fix it better - for now I think we can leave it as it is (based on constraints)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1631555887
> I think for 2.7.0 (hope) when I manage to implement moving celery + kubernetes executors to providers, we might be able to fix it better
I hope so too!
Waiting a response from @vghar-bh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #32484: Celery worker enters a catatonic state after redis restart and the tasks are getting queued
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #32484:
URL: https://github.com/apache/airflow/issues/32484#issuecomment-1630889586
Sure. Good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org