You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Łukasz Pińkowski (Jira)" <ji...@apache.org> on 2019/09/11 09:39:00 UTC
[jira] [Updated] (SPARK-28882) Memory leak when stopping spark
session
[ https://issues.apache.org/jira/browse/SPARK-28882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Łukasz Pińkowski updated SPARK-28882:
-------------------------------------
Description:
When calling stop() method on spark session underlying SparkContext is being stopped.
It causes also stop of underlying ContextCleaner thread, usually before it is able to clean all context objects (not all of them are returned to ReferenceQueue by GC). It causes memory leak because this ReferenceQueue is never collected by GC.
There should be at least comment in documentation that calling stop() method on session or context may lead to memory leaks.
was:
When calling stop() method on spark session underlying SparkContext is being stopped.
It causes also stop of underlying ContextCleaner thread, usually before it is able to clean all context objects (not all of them are returned to ReferenceQueue by GC). It causes memory leak because this ReferenceQueue is never collected by GC.
> Memory leak when stopping spark session
> ---------------------------------------
>
> Key: SPARK-28882
> URL: https://issues.apache.org/jira/browse/SPARK-28882
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.3
> Reporter: Łukasz Pińkowski
> Priority: Major
>
> When calling stop() method on spark session underlying SparkContext is being stopped.
> It causes also stop of underlying ContextCleaner thread, usually before it is able to clean all context objects (not all of them are returned to ReferenceQueue by GC). It causes memory leak because this ReferenceQueue is never collected by GC.
>
> There should be at least comment in documentation that calling stop() method on session or context may lead to memory leaks.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org