Re: Kryo with Java

[Tal Sliwowicz <ta...@taboola.com>]

Hi,

I have a spark job that runs perfectly fine both as standalone and inside a
mesos cluster.

However, when I try to use kryo it fails. In standalone mode it fails
silently and spark gets stuck (nothing happens forever). Inside a cluster,
it gives the message "Failed to register spark.kryo.registrator", but
continues to run.

Do you think that there is a class loader issue here? I've tried adding my
jar to SPARK_CLASSPATH in spark-env.sh, but the issue wasn't resolved.


These are the relevant lines in my code before creating the context:

System.setProperty("spark.serializer",
"org.apache.spark.serializer.KryoSerializer");
System.setProperty("spark.kryo.registrator", MyRegistrator.class.getName());

I have verified that MyRegistrator class isn't loaded using a static {}
section.

Thanks!


Tal