You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Patrick GRANDJEAN <pa...@yahoo.fr.INVALID> on 2017/04/20 14:57:23 UTC
Maximum Partitioner size
Hi,
I have implemented a custom Partitioner (org.apache.spark.Partitioner) that contains a medium-sized object (some megabytes). Unfortunately Spark (2.1.0) fails with a StackOverflowError, and I suspect it is because of the size of the partitioner that needs to be serialized. My question is, what is the maximum size of a Partitioner accepted by Spark?
Thanks!