You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Graham Dennis (JIRA)" <ji...@apache.org> on 2014/08/06 10:21:14 UTC

[jira] [Created] (SPARK-2878) Inconsistent Kryo serialisation with custom Kryo Registrator

Graham Dennis created SPARK-2878:
------------------------------------

             Summary: Inconsistent Kryo serialisation with custom Kryo Registrator
                 Key: SPARK-2878
                 URL: https://issues.apache.org/jira/browse/SPARK-2878
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.0.0
         Environment: Linux RedHat EL 6, 4-node Spark cluster.
            Reporter: Graham Dennis


The custom Kryo Registrator (a class with the org.apache.spark.serializer.KryoRegistrator trait) is not used with every Kryo instance created, and this causes inconsistent serialisation and deserialisation.

The Kryo Registrator is sometimes not used because of a ClassNotFound exception that only occurs if it *isn't* the Worker thread (of an Executor) that tries to create the KryoRegistrator.

A complete description of the problem and a project reproducing the problem can be found at https://github.com/GrahamDennis/spark-kryo-serialisation

I have currently only tested this with Spark 1.0.0, but will try to test against 1.0.2.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org