You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "tgravescs (via GitHub)" <gi...@apache.org> on 2025/05/13 14:34:23 UTC
[D] Spark Driver memory Leak? [kyuubi]
GitHub user tgravescs created a discussion: Spark Driver memory Leak?
Hello,
I am using Kyuubi with Spark on k8s and using python notebooks. We are launching a lot of different queries from multiple different processes over a fairly long period of time and we are seeing the Spark driver memory usage keep increasing. Eventually it starts to GC alot and then OOM.
I got a heap dump of the Spark Driver and it looks like the py4j.Gateway has a concurrentHashMap that is just accumulating objects - Spark Dataset, Atrributes, Expressions, etc. Basically looks like anything that the python process might be referencing on the java side.
From my understanding these are supposed to get cleaned up when the python process does a garbage collection. It seems like that isn't happening here but I'm not positive. I'm wondering if the kyuubi execute_python process is exiting without these being cleaned up.
Has anyone else run into this issue? If so is there any known fixes?
Thanks
GitHub link: https://github.com/apache/kyuubi/discussions/7060
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [D] Spark Driver memory Leak? [kyuubi]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
GitHub user tgravescs added a comment to the discussion: Spark Driver memory Leak?
I've looked at a lot at this the last couple days. The python objects that don't seem to be cleaned up at the ones run in the user code. where execute_python.py does the compile and then exec(). The ones initialized and used in the execute_python.py script tiself seem to be cleaned up ok.
i'm not quite sure why though. I also tried to manually call gc.collect() in the user code without any luck so far. it would be really nice to understand what is going on.
GitHub link: https://github.com/apache/kyuubi/discussions/7060#discussioncomment-13160938
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [D] Spark Driver memory Leak? [kyuubi]
Posted by "HaoYang670 (via GitHub)" <gi...@apache.org>.
GitHub user HaoYang670 added a comment to the discussion: Spark Driver memory Leak?
All python resources are on client side for Spark Connect.
GitHub link: https://github.com/apache/kyuubi/discussions/7060#discussioncomment-13165631
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [D] Spark Driver memory Leak? [kyuubi]
Posted by "HaoYang670 (via GitHub)" <gi...@apache.org>.
GitHub user HaoYang670 added a comment to the discussion: Spark Driver memory Leak?
@pan3793 any insight?
GitHub link: https://github.com/apache/kyuubi/discussions/7060#discussioncomment-13139128
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [D] Spark Driver memory Leak? [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
GitHub user pan3793 added a comment to the discussion: Spark Driver memory Leak?
> ... these are supposed to get cleaned up when the python process does a garbage collection. It seems like that isn't happening ...
Have a quick look at the relative code, the analysis seems right. I'm not familiar with the PySpark stuff, maybe we need to learn something from Connect Server, how does it clean up those resources when a session that accesses Python resources closes?
GitHub link: https://github.com/apache/kyuubi/discussions/7060#discussioncomment-13159725
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org