[GitHub] [airflow] mtraynham commented on issue #21026: Celery Worker docker healthcheck causes a memory leak

[GitBox <gi...@apache.org>]

mtraynham commented on issue #21026:
URL: https://github.com/apache/airflow/issues/21026#issuecomment-1054673312


   @potiuk I've tested Airflow 2.2.4 and still see this issue with the recommended healthcheck.test, https://github.com/apache/airflow/blob/958860fcd7c9ecdf60b7ebeef4397b348835c8db/docs/apache-airflow/start/docker-compose.yaml
   
   I've taken the original Airflow `docker-compose.yml` file and replaced references of `2.2.3` with `2.2.4`.  Also using the included script to collect service stats, from above, I've found the following:
   
   | Date                   | Container                                                 | CPU Percent | Mem Usage           | Mem Percent |
   |------------------------|-----------------------------------------------------------|-------------|---------------------|-------------|
   | 2022-02-28T16:59:44UTC | airflow_worker_no_healthcheck.1.fyihqxhf8hb574ivnz7uflpbz | 2.08%       | 1.123GiB / 14.91GiB | 7.53%       |
   | 2022-02-28T16:59:44UTC | airflow_worker_healthcheck.1.rq48arboysa4b8fqn6i5jb7ad    | 0.43%       | 1.139GiB / 14.91GiB | 7.64%       |
   | 2022-02-28T21:00:41UTC | airflow_worker_no_healthcheck.1.fyihqxhf8hb574ivnz7uflpbz | 0.75%       | 1.123GiB / 14.91GiB | 7.54%       |
   | 2022-02-28T21:00:41UTC | airflow_worker_healthcheck.1.rq48arboysa4b8fqn6i5jb7ad    | 88.68%      | 1.319GiB / 14.91GiB | 8.85%       |
   
   Over a period of 4 hours, memory on the container with the health check has increased ~180 MB in usage, or about 45 MB per hour, which is what I observed before.  Over the course of say 3 days that eventually becomes ~3.24 GB of unclaimed memory.
   
   For reference, the test environment is:
   Airflow 2.2.4 - Celery Executor
   1 Webserver, 1 Scheduler, 2 Workers (1 with health check)
   Ubuntu 20.04.3 LTS
   Linux host1 5.4.0-88-generic 99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
   Docker version 20.10.12, build e91ed57
   
   I've included the dataset as CSV.  Every now and then memory spikes and some is reclaimed by the service, but it still keeps trickling up.
   
   Is there potentially a different health check on each worker that could be used instead?
   
   [stats_airflow.csv](https://github.com/apache/airflow/files/8156896/stats_airflow.csv)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org