You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by jaykatukuri <jk...@apple.com> on 2014/12/09 01:19:27 UTC
MLLib /ALS : java.lang.OutOfMemoryError: Java heap space
Hi all,
I am running into an out of memory error while running ALS using MLLIB on a
reasonably small data set consisting of around 6 Million ratings.
The stack trace is below:
java.lang.OutOfMemoryError: Java heap space
at org.jblas.DoubleMatrix.<init>(DoubleMatrix.java:323)
at org.jblas.DoubleMatrix.zeros(DoubleMatrix.java:471)
at org.jblas.DoubleMatrix.zeros(DoubleMatrix.java:476)
at
org.apache.spark.mllib.recommendation.ALS$$anonfun$21.apply(ALS.scala:465)
at
org.apache.spark.mllib.recommendation.ALS$$anonfun$21.apply(ALS.scala:465)
at scala.Array$.fill(Array.scala:267)
at
org.apache.spark.mllib.recommendation.ALS.org$apache$spark$mllib$recommendation$ALS$$updateBlock(ALS.scala:465)
at
org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateFeatures$2.apply(ALS.scala:445)
at
org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateFeatures$2.apply(ALS.scala:444)
at
org.apache.spark.rdd.MappedValuesRDD$$anonfun$compute$1.apply(MappedValuesRDD.scala:31)
at
org.apache.spark.rdd.MappedValuesRDD$$anonfun$compute$1.apply(MappedValuesRDD.scala:31)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at
org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$4.apply(CoGroupedRDD.scala:156)
at
org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$4.apply(CoGroupedRDD.scala:154)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:154)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedValuesRDD.compute(MappedValuesRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at
org.apache.spark.rdd.FlatMappedValuesRDD.compute(FlatMappedValuesRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.Task.run(Task.scala:51)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
I am using 2GB for executors memory. I tried with 100 executors.
Can some one please point me in the right direction ?
Thanks,
Jay
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLLib-ALS-java-lang-OutOfMemoryError-Java-heap-space-tp20584.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: MLLib /ALS : java.lang.OutOfMemoryError: Java heap space
Posted by Gen <ge...@gmail.com>.
Hi,How many clients and how many products do you have?CheersGen
jaykatukuri wrote
> Hi all,I am running into an out of memory error while running ALS using
> MLLIB on a reasonably small data set consisting of around 6 Million
> ratings.The stack trace is below:java.lang.OutOfMemoryError: Java heap
> space at org.jblas.DoubleMatrix.
> (DoubleMatrix.java:323) at
> org.jblas.DoubleMatrix.zeros(DoubleMatrix.java:471) at
> org.jblas.DoubleMatrix.zeros(DoubleMatrix.java:476) at
> org.apache.spark.mllib.recommendation.ALS$$anonfun$21.apply(ALS.scala:465)
> at
> org.apache.spark.mllib.recommendation.ALS$$anonfun$21.apply(ALS.scala:465)
> at scala.Array$.fill(Array.scala:267) at
> org.apache.spark.mllib.recommendation.ALS.org$apache$spark$mllib$recommendation$ALS$$updateBlock(ALS.scala:465)
> at
> org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateFeatures$2.apply(ALS.scala:445)
> at
> org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateFeatures$2.apply(ALS.scala:444)
> at
> org.apache.spark.rdd.MappedValuesRDD$$anonfun$compute$1.apply(MappedValuesRDD.scala:31)
> at
> org.apache.spark.rdd.MappedValuesRDD$$anonfun$compute$1.apply(MappedValuesRDD.scala:31)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at
> org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$4.apply(CoGroupedRDD.scala:156)
> at
> org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$4.apply(CoGroupedRDD.scala:154)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at
> org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:154) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
> org.apache.spark.rdd.MappedValuesRDD.compute(MappedValuesRDD.scala:31) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
> org.apache.spark.rdd.FlatMappedValuesRDD.compute(FlatMappedValuesRDD.scala:31)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
> org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> at org.apache.spark.scheduler.Task.run(Task.scala:51) at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)I am
> using 2GB for executors memory. I tried with 100 executors.Can some one
> please point me in the right direction ?Thanks,Jay
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLLib-ALS-java-lang-OutOfMemoryError-Java-heap-space-tp20584p20714.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: MLLib /ALS : java.lang.OutOfMemoryError: Java heap space
Posted by buring <qy...@gmail.com>.
you can try to decrease the rank value.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLLib-ALS-java-lang-OutOfMemoryError-Java-heap-space-tp20584p20711.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: MLLib /ALS : java.lang.OutOfMemoryError: Java heap space
Posted by happyyxw <ha...@gmail.com>.
How many working nodes do these 100 executors locate at?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLLib-ALS-java-lang-OutOfMemoryError-Java-heap-space-tp20584p20610.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: MLLib /ALS : java.lang.OutOfMemoryError: Java heap space
Posted by Xiangrui Meng <me...@gmail.com>.
Hi Jay,
Please try increasing executor memory (if the available memory is more
than 2GB) and reduce numBlocks in ALS. The current implementation
stores all subproblems in memory and hence the memory requirement is
significant when k is large. You can also try reducing k and see
whether the problem is still there. I made a PR that improves the ALS
implementation, which generates subproblems one by one. You can try
that as well.
https://github.com/apache/spark/pull/3720
Best,
Xiangrui
On Wed, Dec 17, 2014 at 6:57 PM, buring <qy...@gmail.com> wrote:
> I am not sure this can help you. I have 57 million rating,about 4million user
> and 4k items. I used 7-14 total-executor-cores,executal-memory 13g,cluster
> have 4 nodes,each have 4cores,max memory 16g.
> I found set as follows may help avoid this problem:
> conf.set("spark.shuffle.memoryFraction","0.65") //default is 0.2
> conf.set("spark.storage.memoryFraction","0.3")//default is 0.6
> I have to set rank value under 40, otherwise occure this problem.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLLib-ALS-java-lang-OutOfMemoryError-Java-heap-space-tp20584p20755.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: MLLib /ALS : java.lang.OutOfMemoryError: Java heap space
Posted by buring <qy...@gmail.com>.
I am not sure this can help you. I have 57 million rating,about 4million user
and 4k items. I used 7-14 total-executor-cores,executal-memory 13g,cluster
have 4 nodes,each have 4cores,max memory 16g.
I found set as follows may help avoid this problem:
conf.set("spark.shuffle.memoryFraction","0.65") //default is 0.2
conf.set("spark.storage.memoryFraction","0.3")//default is 0.6
I have to set rank value under 40, otherwise occure this problem.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLLib-ALS-java-lang-OutOfMemoryError-Java-heap-space-tp20584p20755.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org