Unsubscribe

[Alfredo Marquez <al...@gmail.com>]

On Thu, Mar 26, 2020, 8:35 PM Andrew Melo <an...@gmail.com> wrote:

> Hello all,
>
> Is there a way to register classes within a datasourcev2 implementation in
> the Kryo serializer?
>
> I've attempted the following in both the constructor and static block of
> my toplevel class:
>
>         SparkContext context = SparkContext.getOrCreate();
>         SparkConf conf = context.getConf();
>         Class[] classesRegistered = new Class[] {
>                 edu.vanderbilt.accre.laurelin.spark_ttree.Reader.class,
>                 edu.vanderbilt.accre.laurelin.spark_ttree.Partition.class,
>                 edu.vanderbilt.accre.laurelin.spark_ttree.SlimTBranch.class
>                 };
>         conf.registerKryoClasses(classesRegistered);
>
> But (if I'm reading correctly), this is too late, since the config has
> already been parsed while initializing the SparkContext, adding classes to
> the SparkConf has no effect. From what I can tell, the kryo instance behind
> is private, so I can't add the registration manually either.
>
> Any thoughts?
> Thanks
> Andrew
>
>
>