Custom KryoSerializer

[Soren Macbeth <so...@yieldbot.com>]

Does spark support extending and registering a KryoSerializer class in an
application jar?

An example of why you might want to do this would be to always register
some set of common classes within an organization while still allowing the
particular application jar to use a kryo registrator to register additional
serializers for it's own uses.

When I try the naive approach of extending KryoSerializer in my application
jar and registering that class in my SparkConf under the "spark.serializer"
key, my executors all die complaining they can't find my class.

Re: Custom KryoSerializer

[Eugen Cepoi <ce...@gmail.com>]

I had a similar need the solution I used is:
 - Define a base implementation of KryoRegistrator (that will register all
common classes/custom ser/deser)
 - make registerClasses method final, so subclasses don't override it
 - Define another method that would be overriden by subclasses that need to
provide additional customization

So if people need custom config, they just extend your base KryoRegistrator
otherwise they directly use the base impl.


2014-04-23 5:17 GMT+02:00 Soren Macbeth <so...@yieldbot.com>:

> Does spark support extending and registering a KryoSerializer class in an
> application jar?
>
> An example of why you might want to do this would be to always register
> some set of common classes within an organization while still allowing the
> particular application jar to use a kryo registrator to register additional
> serializers for it's own uses.
>
> When I try the naive approach of extending KryoSerializer in my
> application jar and registering that class in my SparkConf under the
> "spark.serializer" key, my executors all die complaining they can't find my
> class.
>