Spark attempts to de/serialize using JavaSerializer despite being configured to use Kryo

[waymost <wa...@gmail.com>]

I'm new to Spark and have run into issues using Kryo for serialization
instead of Java. I have my SparkConf configured as such:

val conf = new SparkConf().setMaster("local").setAppName("test")
        .set("spark.kryo.registrationRequired","false")
        .set("spark.serializer", classOf[KryoSerializer].getName)
        .set("spark.kryo.registrator", classOf[TestRegistrator].getName)

However, when I attempt to run my Spark job, I get the following exception
and stack trace:

java.io.InvalidClassException: ... no valid constructor
at ...
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at ...

If I go into the REPL. use that same SparkConf to create a KryoSerializer
manually, and then do a serialization-deserialization loop on my instance, I
get it returned back to me without an exception.

I'm perplexed as to why Spark isn't using Kryo when running my job. Any help
would be greatly appreciated.

Thanks,
Matthew



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-attempts-to-de-serialize-using-JavaSerializer-despite-being-configured-to-use-Kryo-tp21213.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org