You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Han JU <ju...@gmail.com> on 2013/10/16 12:28:30 UTC
System property for Kryo's setRegistrationRequired
Hi,
I think maybe it's better to have an option to force class registration
when using Kryo, mainly to prevent human errors.
Kryo has a setRegistrationRequired method and by default the check is false
(in kryo and in chill). Maybe we can have a system property for setting
this? Like:
System.setProperty("spark.kryo.registrationRequired", true)
What you say about this?
--
*JU Han*
Data Engineer @ Botify.com
+33 0619608888
Re: System property for Kryo's setRegistrationRequired
Posted by Han JU <ju...@gmail.com>.
Matei, thanks for the reply.
I've tried setting kryo.setRegistrationOptional by KryoRegistrator but it
seems that some spark's internal class, such as
o.a.spark.scheduler.MapStatus, need to be registered.
2013/10/17 Matei Zaharia <ma...@gmail.com>
> This is a good idea, though you may also be able to do it in your
> KryoRegistrator by just calling kryo.setRegistrationOptional there. (You
> get full access to the Kryo object.)
>
> Omitting to register a class is not an error by the way, it just leads to
> bigger data.
>
> Matei
>
> On Oct 16, 2013, at 3:28 AM, Han JU <ju...@gmail.com> wrote:
>
> > Hi,
> >
> > I think maybe it's better to have an option to force class registration
> > when using Kryo, mainly to prevent human errors.
> > Kryo has a setRegistrationRequired method and by default the check is
> false
> > (in kryo and in chill). Maybe we can have a system property for setting
> > this? Like:
> >
> > System.setProperty("spark.kryo.registrationRequired", true)
> >
> > What you say about this?
> > --
> > *JU Han*
> >
> > Data Engineer @ Botify.com
> >
> > +33 0619608888
>
>
--
*JU Han*
Data Engineer @ Botify.com
+33 0619608888
Re: System property for Kryo's setRegistrationRequired
Posted by Matei Zaharia <ma...@gmail.com>.
This is a good idea, though you may also be able to do it in your KryoRegistrator by just calling kryo.setRegistrationOptional there. (You get full access to the Kryo object.)
Omitting to register a class is not an error by the way, it just leads to bigger data.
Matei
On Oct 16, 2013, at 3:28 AM, Han JU <ju...@gmail.com> wrote:
> Hi,
>
> I think maybe it's better to have an option to force class registration
> when using Kryo, mainly to prevent human errors.
> Kryo has a setRegistrationRequired method and by default the check is false
> (in kryo and in chill). Maybe we can have a system property for setting
> this? Like:
>
> System.setProperty("spark.kryo.registrationRequired", true)
>
> What you say about this?
> --
> *JU Han*
>
> Data Engineer @ Botify.com
>
> +33 0619608888