You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Clive Cox <cl...@rummble.com> on 2014/10/09 11:33:27 UTC

DIMSUM item similarity tests

Hi,

 I'm trying out the DIMSUM item similarity from github master commit
69c3f441a9b6e942d6c08afecd59a0349d61cc7b . My matrix is:
Num items : 8860
Number of users : 5138702
Implicit 1.0 values
Running item similarity with threshold :0.5

I have a 2 slave spark cluster on EC2 with m3.xlarge (13G each)

I'm running out of heap space:

Exception in thread "handle-read-write-executor-1"
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
    at org.apache.spark.network.nio.Message$.create(Message.scala:90)

while Spark is doing:

org.apache.spark.rdd.RDD.reduce(RDD.scala:865)
org.apache.spark.mllib.rdd.RDDFunctions.treeAggregate(RDDFunctions.scala:111)
org.apache.spark.mllib.linalg.distributed.RowMatrix.computeColumnSummaryStatistics(RowMatrix.scala:379)
org.apache.spark.mllib.linalg.distributed.RowMatrix.columnSimilarities(RowMatrix.scala:483)

The spark UI said the shuffle read on this task at that point had used:
162.6 MB

I run spark submit from the master like below:

./spark/bin/spark-submit --executor-memory 13G .... --master spark://ec2....

Just wanted to check this is expected as the matrix doesn't seem
excessively big. Is there some memory setting I am missing?

Thanks,

 Clive

Re: DIMSUM item similarity tests

Posted by Xiangrui Meng <me...@gmail.com>.
please re-try with --driver-memory 10g . The default is 256m. -Xiangrui

On Thu, Oct 9, 2014 at 2:33 AM, Clive Cox <cl...@rummble.com> wrote:
> Hi,
>
>  I'm trying out the DIMSUM item similarity from github master commit
> 69c3f441a9b6e942d6c08afecd59a0349d61cc7b . My matrix is:
> Num items : 8860
> Number of users : 5138702
> Implicit 1.0 values
> Running item similarity with threshold :0.5
>
> I have a 2 slave spark cluster on EC2 with m3.xlarge (13G each)
>
> I'm running out of heap space:
>
> Exception in thread "handle-read-write-executor-1"
> java.lang.OutOfMemoryError: Java heap space
>     at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>     at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>     at org.apache.spark.network.nio.Message$.create(Message.scala:90)
>
> while Spark is doing:
>
> org.apache.spark.rdd.RDD.reduce(RDD.scala:865)
> org.apache.spark.mllib.rdd.RDDFunctions.treeAggregate(RDDFunctions.scala:111)
> org.apache.spark.mllib.linalg.distributed.RowMatrix.computeColumnSummaryStatistics(RowMatrix.scala:379)
> org.apache.spark.mllib.linalg.distributed.RowMatrix.columnSimilarities(RowMatrix.scala:483)
>
> The spark UI said the shuffle read on this task at that point had used:
> 162.6 MB
>
> I run spark submit from the master like below:
>
> ./spark/bin/spark-submit --executor-memory 13G .... --master spark://ec2....
>
> Just wanted to check this is expected as the matrix doesn't seem excessively
> big. Is there some memory setting I am missing?
>
> Thanks,
>
>  Clive
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org