You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kalle Jepsen (JIRA)" <ji...@apache.org> on 2015/04/24 09:46:38 UTC
[jira] [Created] (SPARK-7116) Intermediate RDD cached but never
unpersisted
Kalle Jepsen created SPARK-7116:
-----------------------------------
Summary: Intermediate RDD cached but never unpersisted
Key: SPARK-7116
URL: https://issues.apache.org/jira/browse/SPARK-7116
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 1.3.1
Reporter: Kalle Jepsen
In https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUdfs.scala#L233 an intermediate RDD is cached, but never unpersisted. It shows up in the 'Storage' section of the Web UI, but cannot be removed. There's already a comment in the source, suggesting to 'clean up'. If that cleanup is more involved than simply calling `unpersist`, it probably exceeds my current Scala skills.
Why that is a problem:
I'm adding a constant column to a DataFrame of about 20M records resulting from an inner join with {{df.withColumn(colname, ud_func())}} , where {{ud_func}} is simply a wrapped {{lambda: 1}}. Before and after applying the UDF the DataFrame takes up ~430MB in the cache. The cached intermediate RDD however takes up ~10GB(!) of storage, and I know of no way to uncache it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org