You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2019/10/01 02:27:00 UTC
[jira] [Commented] (SPARK-29301) Removing block is not reflected to
the driver/executor's storage memory
[ https://issues.apache.org/jira/browse/SPARK-29301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941473#comment-16941473 ]
Jungtaek Lim commented on SPARK-29301:
--------------------------------------
I'll push the branch-2.4 version of code in my repo and comment. Btw, there's some discussion on how to deal with this issue (and other issue) so the patch may not be shipped as it is. Please refer https://github.com/apache/spark/pull/25779
> Removing block is not reflected to the driver/executor's storage memory
> -----------------------------------------------------------------------
>
> Key: SPARK-29301
> URL: https://issues.apache.org/jira/browse/SPARK-29301
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.4, 3.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> While investigating SPARK-29055 I've found that heap memory in driver doesn't increase which would denote memory leak happen, but storage memory doesn't decrease even there're some broadcast blocks getting removed - these values keep increasing.
> Please refer SPARK-29055 to see the step to reproduce. Very easy to see it, as running simple query repeatedly would reproduce the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org