[jira] [Commented] (SPARK-1418) Python MLlib's _get_unmangled_rdd should uncache RDDs when training is done

["Josh Rosen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323473#comment-14323473 ] 

Josh Rosen commented on SPARK-1418:
-----------------------------------

This issue's description is now a little confusing, since {{_get_unmangled_rdd}} has been removed from the Python MLlib bindings.  Does anyone know if this issue is still relevant?  If so, could we update its description?  Otherwise, let's close this.

> Python MLlib's _get_unmangled_rdd should uncache RDDs when training is done
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-1418
>                 URL: https://issues.apache.org/jira/browse/SPARK-1418
>             Project: Spark
>          Issue Type: Improvement
>          Components: MLlib, PySpark
>            Reporter: Matei Zaharia
>
> Right now when PySpark converts a Python RDD of NumPy vectors to a Java one, it caches the Java one, since many of the algorithms are iterative. We should call unpersist() at the end of the algorithm though to free cache space. In addition it may be good to persist the Java RDD with StorageLevel.MEMORY_AND_DISK instead of going back through the NumPy conversion.. it will almost certainly be faster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org