[jira] [Updated] (SPARK-19504) clearCache fails to delete orphan RDDs, especially in pyspark

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-19504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon updated SPARK-19504:
---------------------------------
    Labels: bulk-closed execution optimizer plan query  (was: execution optimizer plan query)

> clearCache fails to delete orphan RDDs, especially in pyspark
> -------------------------------------------------------------
>
>                 Key: SPARK-19504
>                 URL: https://issues.apache.org/jira/browse/SPARK-19504
>             Project: Spark
>          Issue Type: Bug
>          Components: Optimizer
>    Affects Versions: 2.1.0
>         Environment: Both pyspark and scala spark. Although scala spark uncaches some RDD types even if orphan
>            Reporter: R
>            Priority: Minor
>              Labels: bulk-closed, execution, optimizer, plan, query
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> x=sc.parallelize([1,3,10,9]).cache()
> x.count()
> x=sc.parallelize([1,3,10,9]).cache()
> x.count()
> sqlContex.clearCache()
> Overwriting x will create an orphan RDD, which cannot be deleted with clearCache(). This happens in both scala and pyspark.
> Similar thing happens for rdds created from dataframe in python
> spark.read.csv(....).rdd()
> However, in scala clearCache can get rid of some orphan rdd types.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org