You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Graham Dennis (JIRA)" <ji...@apache.org> on 2014/08/06 10:41:13 UTC
[jira] [Commented] (SPARK-2878) Inconsistent Kryo serialisation
with custom Kryo Registrator
[ https://issues.apache.org/jira/browse/SPARK-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087424#comment-14087424 ]
Graham Dennis commented on SPARK-2878:
--------------------------------------
I've now reproduced this with Spark 1.0.2
> Inconsistent Kryo serialisation with custom Kryo Registrator
> ------------------------------------------------------------
>
> Key: SPARK-2878
> URL: https://issues.apache.org/jira/browse/SPARK-2878
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Environment: Linux RedHat EL 6, 4-node Spark cluster.
> Reporter: Graham Dennis
>
> The custom Kryo Registrator (a class with the org.apache.spark.serializer.KryoRegistrator trait) is not used with every Kryo instance created, and this causes inconsistent serialisation and deserialisation.
> The Kryo Registrator is sometimes not used because of a ClassNotFound exception that only occurs if it *isn't* the Worker thread (of an Executor) that tries to create the KryoRegistrator.
> A complete description of the problem and a project reproducing the problem can be found at https://github.com/GrahamDennis/spark-kryo-serialisation
> I have currently only tested this with Spark 1.0.0, but will try to test against 1.0.2.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org