You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by twinkle sachdeva <tw...@gmail.com> on 2015/04/30 13:13:58 UTC

Regarding KryoSerialization in Spark

Hi,

As per the code, KryoSerialization used writeClassAndObject method, which
internally calls writeClass method, which will write the class of the
object while serilization.

As per the documentation in tuning page of spark, it says that registering
the class will avoid that.

Am I missing something or there is some issue with the documentation???

Thanks,
Twinkle

Re: Regarding KryoSerialization in Spark

Posted by twinkle sachdeva <tw...@gmail.com>.

Thanks for the info.


On Fri, May 1, 2015 at 12:10 AM, Sandy Ryza <sa...@cloudera.com> wrote:

> Hi Twinkle,
>
> Registering the class makes it so that writeClass only writes out a couple
> bytes, instead of a full String of the class name.
>
> -Sandy
>
> On Thu, Apr 30, 2015 at 4:13 AM, twinkle sachdeva <
> twinkle.sachdeva@gmail.com> wrote:
>
>> Hi,
>>
>> As per the code, KryoSerialization used writeClassAndObject method, which
>> internally calls writeClass method, which will write the class of the
>> object while serilization.
>>
>> As per the documentation in tuning page of spark, it says that registering
>> the class will avoid that.
>>
>> Am I missing something or there is some issue with the documentation???
>>
>> Thanks,
>> Twinkle
>>
>
>

Re: Regarding KryoSerialization in Spark

Posted by Sandy Ryza <sa...@cloudera.com>.

Hi Twinkle,

Registering the class makes it so that writeClass only writes out a couple
bytes, instead of a full String of the class name.

-Sandy

On Thu, Apr 30, 2015 at 4:13 AM, twinkle sachdeva <
twinkle.sachdeva@gmail.com> wrote:

> Hi,
>
> As per the code, KryoSerialization used writeClassAndObject method, which
> internally calls writeClass method, which will write the class of the
> object while serilization.
>
> As per the documentation in tuning page of spark, it says that registering
> the class will avoid that.
>
> Am I missing something or there is some issue with the documentation???
>
> Thanks,
> Twinkle
>