You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "VPatel5 (via GitHub)" <gi...@apache.org> on 2026/07/09 14:44:26 UTC
[I] Scheduler Leak [airflow]
VPatel5 opened a new issue, #69660:
URL: https://github.com/apache/airflow/issues/69660
### Under which category would you file this issue?
Helm chart
### Apache Airflow version
3.2.2 (Helm 1.22.0)
### What happened and how to reproduce it?
My airflow scheduler slowly increases in memory usage.
<img width="2129" height="336" alt="Image" src="https://github.com/user-attachments/assets/f7a6b513-5262-4281-89b9-aeb688d2a6b1" />
Below are the flamegraph files of the profiles I ran on the scheduler. The 2nd was a snapshot taken 12+ hours after.
[leaks-flamegraph-1.html](https://github.com/user-attachments/files/29853034/leaks-flamegraph-1.html)
[leaks-flamegraph-2.html](https://github.com/user-attachments/files/29853035/leaks-flamegraph-2.html)
### What you think should happen instead?
Memory usage should stay stable.
### Operating System
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "VPatel5 (via GitHub)" <gi...@apache.org>.
VPatel5 commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-4980057217
**Scheduler 3**
<img width="2239" height="592" alt="Image" src="https://github.com/user-attachments/assets/b419535a-494e-4274-a9c0-a668cf670047" />
As shown in the screenshot, Scheduler 3's memory usage exhibits a clear upward trend, albeit at a very slow and minimal rate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "VPatel5 (via GitHub)" <gi...@apache.org>.
VPatel5 commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-4935732828
> its due to this: [#67127](https://github.com/apache/airflow/pull/67127), [#68905](https://github.com/apache/airflow/pull/68905)
Thanks! Was this fix released in 3.3.0, or is it targeted for a later version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "VPatel5 (via GitHub)" <gi...@apache.org>.
VPatel5 commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-5104040410
Seems like the issue was the profiler. I disabled the profiler configuration, and since then my memory has been stable for 7 days.
Thank you for the help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Leak [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-4926279335
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "lohyenshen (via GitHub)" <gi...@apache.org>.
lohyenshen commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-4967841338
@VPatel5 any updates to your scheduler memory leak?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "VPatel5 (via GitHub)" <gi...@apache.org>.
VPatel5 commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-4938689633
I have updated the celery version to 3.22.0.
<img width="2301" height="148" alt="Image" src="https://github.com/user-attachments/assets/d075844a-66b1-4bde-a929-b447455f9cc5" />
However, it seems the memory is still slowly increasing. It might be too early to tell, so I will provide an update after more observations.
<img width="157" height="244" alt="Image" src="https://github.com/user-attachments/assets/74751a08-34ce-495f-8a8d-da85b95808f9" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-4938321618
> Thanks! Was this fix released in 3.3.0, or is it targeted for a later version?
Released in the Celery provider 3.21.0 version. Let us know if that resolves your issue or if you still observe a memory leak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "VPatel5 (via GitHub)" <gi...@apache.org>.
VPatel5 commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-4972877777
Also, found an observation regarding Memory Usage (Cache) where, in scheduler 3, we are seeing a significantly larger cache size compared to the others.
**Scheduler 1 & 2 (No leak)**
<img width="2237" height="556" alt="Image" src="https://github.com/user-attachments/assets/bb265e3c-1eb0-487e-b1c0-ea0f6f1481f8" />
<img width="2237" height="556" alt="Image" src="https://github.com/user-attachments/assets/34888d7d-a34e-405b-880b-afe87840e6c6" />
**Scheduler 3 (Possible leak)**
<img width="2240" height="557" alt="Image" src="https://github.com/user-attachments/assets/46a5f752-bb67-4a3f-ba7f-fef00845d101" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "VPatel5 (via GitHub)" <gi...@apache.org>.
VPatel5 closed issue #69660: Scheduler Memory Leak
URL: https://github.com/apache/airflow/issues/69660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "arkadiuszbach (via GitHub)" <gi...@apache.org>.
arkadiuszbach commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-4935251458
its due to this: https://github.com/apache/airflow/pull/67127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "VPatel5 (via GitHub)" <gi...@apache.org>.
VPatel5 commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-4969396288
> [@VPatel5](https://github.com/VPatel5) any updates to your scheduler memory leak?
We have observed that the memory usage patterns differ across the three Airflow scheduler replicas over the last 4 days:
**Scheduler 1 (apache-airflow-scheduler-695788959-vvwn4):** Memory grew gradually from ~640 MiB and plateaued at ~1.02 GiB around 07/12. Since then, usage has remained stable.
**Scheduler 2 (apache-airflow-scheduler-695788959-g6t9f):** Memory exhibits frequent, short-lived spikes (roughly every few minutes, each ~15 MB above baseline) with a stable baseline.
**Scheduler 3 (apache-airflow-scheduler-695788959-wdl59):** Memory is continuously and gradually increasing, from ~640 MiB to 1.14 GiB,
**Scheduler 1:**
<img width="2147" height="394" alt="Image" src="https://github.com/user-attachments/assets/04f1d31b-b554-45f9-9687-e1e5e439e0b8" />
**Scheduler 2:**
<img width="2146" height="408" alt="Image" src="https://github.com/user-attachments/assets/96b6554f-6b2b-4fb9-8b12-f86eb19cc2b3" />
**Scheduler 3:**
<img width="2151" height="396" alt="Image" src="https://github.com/user-attachments/assets/c4dd115b-9730-4264-ad4b-a10149127c1a" />
Based on this data, I can't fully conclude that the memory leak is resolved — Scheduler 3 in particular is still trending upward. However, the behavior is definitely improved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler Memory Leak [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on issue #69660:
URL: https://github.com/apache/airflow/issues/69660#issuecomment-5072257166
Would you be able to test it on the latest Airflow version + do a memray? If the memory leak is not from the executor, it will not be patched by the Celery provider. Also, it would be good if you could provide more information regarding your environment, like which executor you use (if you are using Hybrid, the leak may be in a different executor than Celery), dags characteristics, etc., which would make it easier for other people to possibly replicate the behaviour and help with the investigation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org