Re: [I] Small memory leak in worker & scheduler (2.10.x) or dag-processor (3.0.x) [airflow]

["zachliu (via GitHub)" <gi...@apache.org>]

zachliu commented on issue #50708:
URL: https://github.com/apache/airflow/issues/50708#issuecomment-2904785792

   @tirkarthi i think have identified at least two leaks and they don't seem to show up in fd:
   1. caused by these two library imports. this leak majorly affects the `worker`. it has little to no impact in other services unless you're using `["CMD-SHELL", 'airflow ...'] ` in docker-compose `healthcheck`
       ```python
       from importlib import metadata
       from sqlalchemy import create_engine
       ```
   
       to reproduce, run this in any airflow docker container
       ```bash
       #!/bin/bash
   
       while true; do
         # both of the following can show the leak
         # airflow version
         python test.py  # test.py contains the above two imports
         sleep 0.5
       done
       ```
   
   2. (possibly) caused by certain sub-process manage mechanism in `dag-processor`. when there are no dags, you can't see this leak. to reproduce i duplicate this [test_bash_operator.py.txt](https://github.com/user-attachments/files/20412998/test_bash_operator.py.txt) 750 times each with a unique dag name (`test_bash_operator_X`). then you can see `dag-processor` memory usage creeping up
   
   right now i have no idea whether these 2 leaks are related or not. here is my test environment:
   1. airflow 2.10.5 and 3.0.1. local `docker compose` with `CeleryExecutor`
   2. no example dags no other dags but the 750 [test_bash_operator.py.txt](https://github.com/user-attachments/files/20412998/test_bash_operator.py.txt) DAGs
   3. no plugins
   4. no custom logging configs
   5. no custom `webserver_config.py`
   6. use `SimpleAuthManager`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org