[GitHub] [airflow] MrManicotti commented on pull request #19703: upgrade celery 5.2.3

[GitBox <gi...@apache.org>]

MrManicotti commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1018778222


   FYI this version increase includes some much needed fixes regarding memory leaks in Celery w/ Redis brokers. We were experiencing our airflow workers failing from memory leaks due to health checks - after some further investigation it seems due to https://github.com/celery/celery/issues/4843.
   
   Pretty good description of the issue in https://github.com/celery/celery/issues/4843#issuecomment-991394349
   
   For our side, a fix was made in **Kombo 5.2.3** and included in **Celery 5.2.3** which resolve the memory leak issue:
   https://github.com/celery/kombu/pull/1476 to resolve the leaking `Transport` objects that stayed in mem from closed connections
   
   A change was also included to shrink the overall memory usage of `Transport` and `Connection` objects in case of further memory leaks that were missed. https://github.com/celery/kombu/pull/1470
   
   For an overall review of what was changed to resolve the memory leaks, please see https://github.com/celery/celery/issues/4843#issuecomment-1000657871
   
   I'm not sure regarding your release cadence, but this includes some pretty big fixes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org