You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "prayaagmahajan (via GitHub)" <gi...@apache.org> on 2026/02/09 19:50:38 UTC
[D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
GitHub user prayaagmahajan created a discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
I am observing a consistent memory leak in our environment for components(Webserver, Scheduler, Celery). Memory consumption increases steadily over time and does not return to baseline levels, even when the system is idle or after the completion of heavy DAG runs.
Airflow Version: 3.1.7 (recently migrated from 2.10.3)
Executor: [CeleryExecutor]
Python Version: [3.11.3]
OS/Container Image: [Ubuntu]
PS: I am looking a steep memory consumption highly in webserver. Is it by any chance due to some FastAPI configuration which is not releasing the memory?
Please let me know if you need more information also one more thing I would like to add i.e how can I fine tune my celery and scheduler to work more efficiently and complete task execution quick and effective is there any configs I can try out related to this performance part? The reason I asked this is when I was using 3.1.3 my dag was getting executed in 9 mins and with 3.1.7 its taking around 14 mins of time
Attaching some metrics for better understanding
<img width="589" height="210" alt="image" src="https://github.com/user-attachments/assets/5183dcfa-eea7-43fa-921e-39c36bc2c078" />
<img width="601" height="215" alt="image" src="https://github.com/user-attachments/assets/c15dfc15-b168-4d61-b17d-3c8ccd616a46" />
<img width="597" height="206" alt="image" src="https://github.com/user-attachments/assets/dce3ad4b-899f-461a-b0ae-89123ac6a809" />
GitHub link: https://github.com/apache/airflow/discussions/61684
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "yevsh (via GitHub)" <gi...@apache.org>.
GitHub user yevsh added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
Is there a ETA to this fix, we also have MEM issue in api-server 3.1.5
(pod restarts , 2gb mem)
<img width="1612" height="356" alt="image" src="https://github.com/user-attachments/assets/03ebce71-60a4-476b-8367-b8d9f67db95b" />
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15825108
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "prayaagmahajan (via GitHub)" <gi...@apache.org>.
GitHub user prayaagmahajan deleted a comment on the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
Hi Airflow Team, I really need your help here to understand how can I fix this leak within my components!
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15764194
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "prayaagmahajan (via GitHub)" <gi...@apache.org>.
GitHub user prayaagmahajan edited a comment on the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
Hi @wjddn279 , I am running SQL task within the NESTED PIPELINES. so overall this is the setup we have a MASTER PIPELINE and few NESTED PIPELINES under it and under NESTED we have few TASK pipelines and each TASK PIPELINE have few SQL TASKS.
yes by execution time I meant running time.
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15780710
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "prayaagmahajan (via GitHub)" <gi...@apache.org>.
GitHub user prayaagmahajan added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
Hi @wjddn279 , The Dag file I can't share but I am running SQL task within the NESTED PIPELINES. so overall this is the setup we have a MASTER PIPELINE and few NESTED PIPELINES under it and under NESTED we have few TASK pipelines and each TASK PIPELINE have few SQL TASKS.
yes by execution time I meant running time.
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15780710
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
There were some fixes in 3.1.7 addressing memory. First check this one - more coming in 3.1.8 I believe. No ETAs -we release when ready and we expect people who can upgrade to later versions to upgrade to those.
Is there an ETA on your upgrade @yevsh ?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15829415
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
GitHub user wjddn279 added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
@prayaagmahajan
> The reason I asked this is when I was using 3.1.3 my dag was getting executed in 9 mins and with 3.1.7 its taking around 14 mins of time
Interesting point. Can you provide more information like task what you run, dag file, etc.. ?
And i want to know if the executed time means task running time?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15778149
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "prayaagmahajan (via GitHub)" <gi...@apache.org>.
GitHub user prayaagmahajan edited a comment on the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
@wjddn279
1. What is version of apache-airflow-providers-celery you use before and after?:- I am using 'apache-airflow-providers-celery==3.15.2'
2. do you want me to try and use 3.13.0 for celery provider with airflow 3.1.7?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15814451
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
GitHub user wjddn279 added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
yes. exactly
And can you provide the dag file? it would be great resource to resolve the issue for another users
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15820454
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
GitHub user wjddn279 edited a comment on the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
@prayaagmahajan
Could you please check this?
1. What is version of apache-airflow-providers-celery you use before and after?
2. If you patch the version of apache-airflow-providers-celery to 3.13.0 in airflow 3.1.7 image, the performance degradation still occurs?
To accurately diagnose the problem, I need the complete DAG file you used. Could you provide the DAG file with sensitive information (queries, connection details, etc.) removed?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15809417
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "yevsh (via GitHub)" <gi...@apache.org>.
GitHub user yevsh added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
Does anyone can confirm if this is fixed in 3.1.8?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-16138754
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "lohyenshen (via GitHub)" <gi...@apache.org>.
GitHub user lohyenshen edited a comment on the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
@potiuk
what if it's the scheduler that's memory leaking?
im running on airflow 3.2.2, with helm-chart/1.22.0.
executor=CeleryExecutor
my scheduler has memory leak
<img width="723" height="397" alt="image" src="https://github.com/user-attachments/assets/94e84af3-020b-4610-a568-05a6f63346d7" />
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-17558999
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "anavrotski (via GitHub)" <gi...@apache.org>.
GitHub user anavrotski added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
In Airflow 3.1.6 I'm observing memory leak in api-server only, all other pods looks normal
<img width="691" height="345" alt="Screenshot 2026-02-10 at 10 10 21" src="https://github.com/user-attachments/assets/2a24cb11-bbf3-4872-a389-d149e36b840d" />
<img width="691" height="345" alt="Screenshot 2026-02-10 at 10 10 55" src="https://github.com/user-attachments/assets/236cf98f-dcb0-4ff2-a425-efb6d44723fd" />
<img width="691" height="345" alt="Screenshot 2026-02-10 at 10 13 17" src="https://github.com/user-attachments/assets/bb2a9d81-dca0-4150-8e13-884a2a312961" />
<img width="691" height="345" alt="Screenshot 2026-02-10 at 10 13 46" src="https://github.com/user-attachments/assets/c9a70cb6-c9d8-4f9b-a0b7-3fbf5a24d804" />
<img width="691" height="345" alt="Screenshot 2026-02-10 at 10 14 08" src="https://github.com/user-attachments/assets/b6a71ac5-f61c-4fb6-ae91-3869ff2c4061" />
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15753381
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "prayaagmahajan (via GitHub)" <gi...@apache.org>.
GitHub user prayaagmahajan added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
Hi @wjddn279 , We are submitting queries to a MySQL engine via a wrapper. The 'execution time' we are tracking is the total time the task process stays active in the pipeline, which includes the connection handoff and the actual SQL execution.
And Yes there is no difference to each query execution time just by executing the query!
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15785316
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
Open issue - but do tests with memray first. https://airflow.apache.org/docs/apache-airflow/stable/howto/memory-profiling.html
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-17559489
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "yevsh (via GitHub)" <gi...@apache.org>.
GitHub user yevsh added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
3.1.7
<img width="1478" height="332" alt="image" src="https://github.com/user-attachments/assets/b414bf09-6d3a-4e82-978a-1ba4a2a0a52b" />
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15845117
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "prayaagmahajan (via GitHub)" <gi...@apache.org>.
GitHub user prayaagmahajan added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
Hi Airflow Team, I really need your help here to understand how can I fix this leak within my components!
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15764194
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "lohyenshen (via GitHub)" <gi...@apache.org>.
GitHub user lohyenshen added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
@potiuk
this is my memray flamegraph, tho I'm not entirely sure what's causing memory leak in scheduler.
[memray-flamegraph-airflow-scheduler-6d46cc7d7d-fhhhq_scheduler_memory.html](https://github.com/user-attachments/files/30001291/memray-flamegraph-airflow-scheduler-6d46cc7d7d-fhhhq_scheduler_memory.html)
Do you mind to help take a quick look?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-17633894
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "trina242 (via GitHub)" <gi...@apache.org>.
GitHub user trina242 added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
I upgraded yesterday to 3.1.8, not solved. Taking a closer look it seems that memory is increasing on certain events, I'll try to look for a root cause.
<img width="1512" height="794" alt="Screenshot 2026-03-26 at 11 59 38" src="https://github.com/user-attachments/assets/cd201bf7-7550-4b6a-9080-757ebdf28aab" />
<img width="1512" height="796" alt="Screenshot 2026-03-26 at 12 00 10" src="https://github.com/user-attachments/assets/7ac2c98e-9fb4-4a8a-a013-174c3fdd3021" />
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-16325366
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
GitHub user wjddn279 edited a comment on the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
@prayaagmahajan
To accurately diagnose the problem, I need the complete DAG file you used. Could you provide the DAG file with sensitive information (queries, connection details, etc.) removed?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15809417
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "yevsh (via GitHub)" <gi...@apache.org>.
GitHub user yevsh added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
can anyone confirm if was fixed in 3.2.1?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-16751941
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
GitHub user wjddn279 added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
What is version of `apache-airflow-providers-celery` you use before and after?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15809417
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
GitHub user wjddn279 added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
what is query engine you submit the query? there is no difference to each query execution time?
Just only executing the query and don't do anything else?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15780803
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
I think you can chime in in the PR here https://github.com/apache/airflow/pull/60804 -> there are some changes that will address ever-growing memory of the api-server. For now you can easily workaround it by configuring your deployment to have multiple api-servers and gracefully shutting eah of them down periodically. Api-servers are stateless, so you can have more of them and have load-balancing in front and restart some of the servers while others will be serving traffic.
With the PR and other changs discussed we are likely to have it easier/more automated, but for now you have to handle it on your own.
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15769238
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "Jeoffreybauvin (via GitHub)" <gi...@apache.org>.
GitHub user Jeoffreybauvin added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
Not solved, indeed :(.
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-16327474
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "lohyenshen (via GitHub)" <gi...@apache.org>.
GitHub user lohyenshen deleted a comment on the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
@potiuk
this is my memray flamegraph, tho I'm not entirely sure what's causing memory leak in scheduler.
[memray-flamegraph-airflow-scheduler-6d46cc7d7d-fhhhq_scheduler_memory.html](https://github.com/user-attachments/files/30001291/memray-flamegraph-airflow-scheduler-6d46cc7d7d-fhhhq_scheduler_memory.html)
Do you mind to help take a quick look?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-17633894
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
GitHub user wjddn279 edited a comment on the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
@prayaagmahajan
Could you please check this?
1. What is version of `apache-airflow-providers-celery` you use before and after?
2. If you patch the version of `apache-airflow-providers-celery` to 3.13.0 in airflow 3.1.7 image, the performance degradation still occurs?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15809417
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "prayaagmahajan (via GitHub)" <gi...@apache.org>.
GitHub user prayaagmahajan added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
@wjddn279
1. What is version of apache-airflow-providers-celery you use before and after?:- I am using 'apache-airflow-providers-celery==3.15.2'
2. do you want me to try and use 3.13.0 for celery provider?
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-15814451
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "anavrotski (via GitHub)" <gi...@apache.org>.
GitHub user anavrotski added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
And these restarts of the API server are the reason for this error in Audit Log
```
Executor CeleryExecutor(parallelism=32) reported that the task instance <TaskInstance: ... [running]> finished
with state failed, but the task instance's state attribute is running.
Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally
```
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-16131922
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "Jeoffreybauvin (via GitHub)" <gi...@apache.org>.
GitHub user Jeoffreybauvin added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
One question : are you using xcoms with database storage ? Since I migrated xcoms to S3 storage, the memory consumption seems to be flat...
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-16329218
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler [airflow]
Posted by "lohyenshen (via GitHub)" <gi...@apache.org>.
GitHub user lohyenshen added a comment to the discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
@potiuk
what if it's the scheduler that's memory leaking?
im running on airflow 3.2.2, with helm-chart/1.22.0.
executor=CeleryExecutor
<img width="723" height="397" alt="image" src="https://github.com/user-attachments/assets/94e84af3-020b-4610-a568-05a6f63346d7" />
GitHub link: https://github.com/apache/airflow/discussions/61684#discussioncomment-17558999
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org