You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@systemml.apache.org by "Matthias Boehm (JIRA)" <ji...@apache.org> on 2017/04/19 01:18:41 UTC
[jira] [Commented] (SYSTEMML-1543) Parfor remote result merge fails
w/ stackoverflowerror from rdd union
[ https://issues.apache.org/jira/browse/SYSTEMML-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973845#comment-15973845 ]
Matthias Boehm commented on SYSTEMML-1543:
------------------------------------------
cc [~acs_s]
> Parfor remote result merge fails w/ stackoverflowerror from rdd union
> ---------------------------------------------------------------------
>
> Key: SYSTEMML-1543
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1543
> Project: SystemML
> Issue Type: Bug
> Reporter: Matthias Boehm
> Priority: Blocker
>
> The current parfor remote_spark result merge is implemented as (1) a union over all output files from individual parfor tasks, and (2) a final aggregation with or without compare. For many parfor tasks, the union of such many RDDs leads to stack overflow errors as shown below:
> {code}
> Exception in thread "main" java.lang.StackOverflowError
> at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84)
> at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.immutable.List.map(List.scala:285)
> at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:84)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
> at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84)
> at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.immutable.List.foreach(List.scala:381)
> {code}
> This did not show up before because the parfor optimizer usually compiles in-memory result merge operations if the outputs are moderately small, i.e., they fit into the driver.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)