You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chao Fang (JIRA)" <ji...@apache.org> on 2018/08/28 13:40:00 UTC
[jira] [Comment Edited] (SPARK-25091) UNCACHE TABLE, CLEAR CACHE,
rdd.unpersist() does not clean up executor memory
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594981#comment-16594981 ]
Chao Fang edited comment on SPARK-25091 at 8/28/18 1:39 PM:
------------------------------------------------------------
Yes, I think it's UI issue.
Today I run the CACHE/UNCACHE TABLE for three times and finally REFRESH TABLE. And I get the attached files. As you can see, the storage tab is ok， while Storage Memory in Executor Tab always increase. And as you can see, the old gen space release memory space as expected and the Disk Memory in Executor Tab is 0.0B
was (Author: chao fang):
Yes, I think it's UI issue.
Today I run the CACHE/UNCACHE TABLE for three times and finally REFRESH TABLE. And I get the attached files. As you can see, the storage tab is ok， while Storage Memory in Executor Tab always increase. And as you can see, the old gen space release memory space as expected and the Disk Memory in Executor Tab is 0.0B !4.png!
> UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory
> -----------------------------------------------------------------------------
>
> Key: SPARK-25091
> URL: https://issues.apache.org/jira/browse/SPARK-25091
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.1
> Reporter: Yunling Cai
> Priority: Critical
> Attachments: 0.png, 1.png, 2.png, 3.png, 4.png
>
>
> UNCACHE TABLE and CLEAR CACHE does not clean up executor memory.
> Through Spark UI, although in Storage, we see the cached table removed. In Executor, the executors continue to hold the RDD and the memory is not cleared. This results in huge waste in executor memory usage. As we call CACHE TABLE, we run into issues where the cached tables are spilled to disk instead of reclaiming the memory storage.
> Steps to reproduce:
> CACHE TABLE test.test_cache;
> UNCACHE TABLE test.test_cache;
> == Storage shows table is not cached; Executor shows the executor storage memory does not change ==
> CACHE TABLE test.test_cache;
> CLEAR CACHE;
> == Storage shows table is not cached; Executor shows the executor storage memory does not change ==
> Similar behavior when using pyspark df.unpersist().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org