You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/20 14:04:01 UTC
[GitHub] [airflow] theophanevie commented on issue #15799: celery worker timed out with rabbitmq
theophanevie commented on issue #15799:
URL: https://github.com/apache/airflow/issues/15799#issuecomment-1285605202
Having this issue on airflow `2.4.1` aswell
celery version 5.2.7
celery configuration :
``` python
celery_app.conf.update(
worker_prefetch_multiplier=1,
task_acks_late=True,
task_reject_on_worker_lost=True,
worker_max_tasks_per_child=100,
worker_cancel_long_running_tasks_on_connection_loss=True,
)
```
worker log :
``` shell
[2022-10-20T06:16:28.195699315Z] ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2396)
--
| | [2022-10-20T06:16:28.195695965Z] return self._sslobj.write(data)
| | [2022-10-20T06:16:28.195692435Z] File "/usr/local/lib/python3.10/ssl.py", line 1149, in write
| | [2022-10-20T06:16:28.195689285Z] n = write(s)
| | [2022-10-20T06:16:28.195685375Z] File "/usr/local/lib/python3.10/site-packages/amqp/transport.py", line 595, in _write
| | [2022-10-20T06:16:28.195681955Z] self._write(s)
| | [2022-10-20T06:16:28.195678275Z] File "/usr/local/lib/python3.10/site-packages/amqp/transport.py", line 347, in write
| | [2022-10-20T06:16:28.195674215Z] write(buffer_store.view[:offset])
| | [2022-10-20T06:16:28.195670455Z] File "/usr/local/lib/python3.10/site-packages/amqp/method_framing.py", line 186, in write_frame
| | [2022-10-20T06:16:28.195667145Z] conn.frame_writer(1, self.channel_id, sig, args, content)
| | [2022-10-20T06:16:28.195663465Z] File "/usr/local/lib/python3.10/site-packages/amqp/abstract_channel.py", line 70, in send_method
| | [2022-10-20T06:16:28.195658615Z] return self.send_method(
| | [2022-10-20T06:16:28.195655015Z] File "/usr/local/lib/python3.10/site-packages/amqp/channel.py", line 1407, in basic_ack
| | [2022-10-20T06:16:28.195651455Z] self.channel.basic_ack(self.delivery_tag, multiple=multiple)
| | [2022-10-20T06:16:28.195643425Z] File "/usr/local/lib/python3.10/site-packages/kombu/message.py", line 123, in ack
| | [2022-10-20T06:16:28.195640275Z] self.ack(multiple=multiple)
| | [2022-10-20T06:16:28.195636695Z] File "/usr/local/lib/python3.10/site-packages/kombu/message.py", line 128, in ack_log_error
| | [2022-10-20T06:16:28.195633284Z] Traceback (most recent call last):
| | [2022-10-20T06:16:28.195630094Z]
| | [2022-10-20T06:16:28.195626734Z] During handling of the above exception, another exception occurred:
| | [2022-10-20T06:16:28.195623684Z]
| | [2022-10-20T06:16:28.195619844Z] amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
| | [2022-10-20T06:16:28.195616414Z] raise error_for_code(
| | [2022-10-20T06:16:28.195612414Z] File "/usr/local/lib/python3.10/site-packages/amqp/channel.py", line 293, in _on_close
| | [2022-10-20T06:16:28.195609064Z] listener(*args)
| | [2022-10-20T06:16:28.195604304Z] File "/usr/local/lib/python3.10/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
| | [2022-10-20T06:16:28.195600954Z] return self.channels[channel_id].dispatch_method(
| | [2022-10-20T06:16:28.195597284Z] File "/usr/local/lib/python3.10/site-packages/amqp/connection.py", line 537, in on_inbound_method
| | [2022-10-20T06:16:28.195593984Z] callback(channel, method_sig, buf, None)
| | [2022-10-20T06:16:28.195590474Z] File "/usr/local/lib/python3.10/site-packages/amqp/method_framing.py", line 53, in on_frame
| | [2022-10-20T06:16:28.195587124Z] return self.on_inbound_frame(frame)
| | [2022-10-20T06:16:28.195583494Z] File "/usr/local/lib/python3.10/site-packages/amqp/connection.py", line 531, in blocking_read
| | [2022-10-20T06:16:28.195580234Z] while not self.blocking_read(timeout):
| | [2022-10-20T06:16:28.195576514Z] File "/usr/local/lib/python3.10/site-packages/amqp/connection.py", line 525, in drain_events
| | [2022-10-20T06:16:28.195573244Z] drain_events(timeout=0)
| | [2022-10-20T06:16:28.195569814Z] File "/usr/local/lib/python3.10/site-packages/kombu/transport/base.py", line 217, in _read
| | [2022-10-20T06:16:28.195566844Z] reader(loop)
| | [2022-10-20T06:16:28.195563264Z] File "/usr/local/lib/python3.10/site-packages/kombu/transport/base.py", line 235, in on_readable
| | [2022-10-20T06:16:28.195560104Z] cb(*cbargs)
| | [2022-10-20T06:16:28.195556484Z] File "/usr/local/lib/python3.10/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
| | [2022-10-20T06:16:28.195553264Z] next(loop)
| | [2022-10-20T06:16:28.195549584Z] File "/usr/local/lib/python3.10/site-packages/celery/worker/loops.py", line 97, in asynloop
| | [2022-10-20T06:16:28.195546274Z] c.loop(*c.loop_args())
| | [2022-10-20T06:16:28.195542404Z] File "/usr/local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 628, in start
| | [2022-10-20T06:16:28.195530283Z] step.start(parent)
| | [2022-10-20T06:16:28.195526623Z] File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
| | [2022-10-20T06:16:28.195523193Z] blueprint.start(self)
| | [2022-10-20T06:16:28.195519783Z] File "/usr/local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 332, in start
| | [2022-10-20T06:16:28.195516343Z] return self.obj.start()
| | [2022-10-20T06:16:28.195512243Z] File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start
| | [2022-10-20T06:16:28.195488873Z] step.start(parent)
| | [2022-10-20T06:16:28.195485193Z] File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
| | [2022-10-20T06:16:28.195481333Z] self.blueprint.start(self)
| | [2022-10-20T06:16:28.195477453Z] File "/usr/local/lib/python3.10/site-packages/celery/worker/worker.py", line 203, in start
| | [2022-10-20T06:16:28.195473203Z] Traceback (most recent call last):
| | [2022-10-20T06:16:28.195455983Z] [2022-10-20 06:16:28,193: CRITICAL/MainProcess] Couldn't ack 6, reason:SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2396)')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org