You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Anand Avati <av...@gluster.org> on 2014/05/12 23:47:03 UTC
Kryo not default?
Hi,
Can someone share the reason why Kryo serializer is not the default? Is
there anything to be careful about (because of which it is not enabled by
default)?
Thanks!
Re: Kryo not default?
Posted by Matei Zaharia <ma...@gmail.com>.
It was just because it might not work with some user data types that are Serializable. But we should investigate it, as it’s the easiest thing one can enable to improve performance.
Matei
On May 12, 2014, at 2:47 PM, Anand Avati <av...@gluster.org> wrote:
> Hi,
> Can someone share the reason why Kryo serializer is not the default? Is
> there anything to be careful about (because of which it is not enabled by
> default)?
> Thanks!
Re: Kryo not default?
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
On Mon, May 12, 2014 at 2:47 PM, Anand Avati <av...@gluster.org> wrote:
> Hi,
> Can someone share the reason why Kryo serializer is not the default?
why should it be?
On top of it, the only way to serialize a closure into the backend (even
now) is java serialization (which means java serialization is required of
all closure attributes)
> Is
> there anything to be careful about (because of which it is not enabled by
> default)?
>
Yes. Kind of stems from above. There's still a number of api calls that use
closure attributes to serialize data to backend (see fold(), for example).
which means even if you enable kryo, some api still requires java
serialization of an attribute.
I fixed parallelize(), collect() and something else that i don't remember
already in that regard, but i think even up till now there's still a number
of apis lingering whose data parameters wouldn't work with kryo.
> Thanks!
>
Re: Kryo not default?
Posted by Andrew Ash <an...@andrewash.com>.
As an example of where it sometimes doesn't work, in older versions of Kryo
/ Chill the Joda LocalDate class didn't serialize properly --
https://groups.google.com/forum/#!topic/cascalog-user/35cdnNIamKU
On Mon, May 12, 2014 at 4:39 PM, Reynold Xin <rx...@databricks.com> wrote:
> The main reason is that it doesn't always work (e.g. sometimes application
> program has special serialization / externalization written already for
> Java which don't work in Kryo).
>
> On Mon, May 12, 2014 at 5:47 PM, Anand Avati <av...@gluster.org> wrote:
>
> > Hi,
> > Can someone share the reason why Kryo serializer is not the default? Is
> > there anything to be careful about (because of which it is not enabled by
> > default)?
> > Thanks!
> >
>
Re: Kryo not default?
Posted by Reynold Xin <rx...@databricks.com>.
The main reason is that it doesn't always work (e.g. sometimes application
program has special serialization / externalization written already for
Java which don't work in Kryo).
On Mon, May 12, 2014 at 5:47 PM, Anand Avati <av...@gluster.org> wrote:
> Hi,
> Can someone share the reason why Kryo serializer is not the default? Is
> there anything to be careful about (because of which it is not enabled by
> default)?
> Thanks!
>