You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by akhandeshi <am...@gmail.com> on 2014/11/17 19:31:09 UTC

Re: java.lang.OutOfMemoryError: Requested array size exceeds VM limit

"only option is to split you problem further by increasing parallelism" My
understanding is  by increasing the number of partitions, is that right? 
That didn't seem to help because it is seem the partitions are not uniformly
sized.   My observation is when I increase the number of partitions, it
creates many "empty block partitions" and may larger partition is not broken
down into smaller size.  Any hints, on how I can get uniform partitions.  I
noticed many threads, but was not able to do any thing effective from Java
api.  I will appreciate any help/insight you can provide.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-OutOfMemoryError-Requested-array-size-exceeds-VM-limit-tp16809p19097.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: java.lang.OutOfMemoryError: Requested array size exceeds VM limit

Posted by nir <ni...@gmail.com>.
For uniform partitioning, you can try custom Partitioner. 



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-OutOfMemoryError-Requested-array-size-exceeds-VM-limit-tp16809p26477.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org