[jira] [Commented] (SPARK-2141) Add sc.getPersistentRDDs() to PySpark

["Nicholas Chammas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824360#comment-15824360 ] 

Nicholas Chammas commented on SPARK-2141:
-----------------------------------------

I'd like to reopen this issue given the fact that the Scala and [Java|http://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#getPersistentRDDs()] APIs have this, and that several people have commented saying they would find this useful.

I'll add my own use case to the list.

In the course of working around various Spark bugs that are mitigated by strategically placed calls to {{persist()}} ([ex1|https://github.com/graphframes/graphframes/issues/159], [ex2|https://issues.apache.org/jira/browse/SPARK-18492]), it's useful to be able to unpersist all RDDs, or all RDDs that you don't have active references to, to free up memory. As others have noted, you can do this today by dipping into the Java context:

{code}
for (id, rdd) in spark.sparkContext._jsc.getPersistentRDDs().items():
    rdd.unpersist()
{code}

But I think it makes sense to add this directly to the Python API.

> Add sc.getPersistentRDDs() to PySpark
> -------------------------------------
>
>                 Key: SPARK-2141
>                 URL: https://issues.apache.org/jira/browse/SPARK-2141
>             Project: Spark
>          Issue Type: New Feature
>          Components: PySpark
>    Affects Versions: 1.0.0
>            Reporter: Nicholas Chammas
>            Assignee: Kan Zhang
>
> PySpark does not appear to have {{sc.getPersistentRDDs()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org