[jira] [Commented] (SPARK-15060) Fix stack overflow when executing long lineage transform without checkpoint

["Michael Wu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16849732#comment-16849732 ] 

Michael Wu commented on SPARK-15060:
------------------------------------

I'm late to the party but have you tried using `localCheckpoint` instead of `checkpoint`? It _should_ be faster at the cost of not being fault-tolerant in the case of executor failure / pre-emption.

> Fix stack overflow when executing long lineage transform without checkpoint
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-15060
>                 URL: https://issues.apache.org/jira/browse/SPARK-15060
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.5.2, 1.6.1, 2.0.0
>            Reporter: Zheng Tan
>            Priority: Major
>              Labels: bulk-closed
>
> When executing long linage rdd transform, it is easy to get stack overflow exception in driver end. This can be reproduced by the following example:
> var rdd = sc.makeRDD(1 to 10, 10)
> for (_ <- 1 to 1000) {
>   rdd = rdd.map(x => x)
> }
> rdd.reduce(_ + _)
> SPARK-5955 solve this problem by checkpointing rdd for every 10~20 rounds. It is not so convenient since it required checkpointing data to HDFS. 
> Another solution is cutting off the recursive rdd dependencies in driver end and re-assembly them in executor end.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org