You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by anny9699 <an...@gmail.com> on 2014/10/03 20:28:30 UTC

array size limit vs partition number

Hi,

Sorry I am not very familiar with Java. I found that if I set the RDD
partition number to be higher, I meet this error
message"java.lang.OutOfMemoryError: Requested array size exceeds VM limit";
however if I set the RDD partition number to be lower, the error is gone.

My aws ec2 cluster has 72 cores, so I first set the partition number to be
150, and met the above problem. Then I set the partition number to be 100,
the error is gone.

Could anybody explain?

Thanks a lot!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/array-size-limit-vs-partition-number-tp15695.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org