You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Koert Kuipers <ko...@tresata.com> on 2017/01/19 21:17:07 UTC
dataset aggregators with kryo encoder very slow
we just converted a job from RDD to Dataset. the job does a single map-red
phase using aggregators. we are seeing very bad performance for the Dataset
version, about 10x slower.
in the Dataset version we use kryo encoders for some of the aggregators.
based on some basic profiling of spark in local mode i believe the bad
performance is due to the kryo encoders. about 70% of time is spend in kryo
related classes.
since we also use kryo for serialization with the RDD i am surprised how
big the performance difference is.
has anyone seen the same thing? any suggestions for how to improve this?
Re: dataset aggregators with kryo encoder very slow
Posted by Koert Kuipers <ko...@tresata.com>.
sorry i meant to say SPARK-18980
On Sat, Jan 21, 2017 at 1:48 AM, Koert Kuipers <ko...@tresata.com> wrote:
> found it :) SPARK-1890
> thanks cloud-fan
>
> On Sat, Jan 21, 2017 at 1:46 AM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> trying to replicate this in spark itself i can for v2.1.0 but not for
>> master. i guess it has been fixed
>>
>> On Fri, Jan 20, 2017 at 4:57 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> i started printing out when kryo serializes my buffer data structure for
>>> my aggregator.
>>>
>>> i would expect every buffer object to ideally get serialized only once:
>>> at the end of the map-side before the shuffle (so after all the values for
>>> the given key within the partition have been reduced into it). i realize
>>> that in reality due to the order of the elements coming in this can not
>>> always be achieved. but what i see instead is that the buffer is getting
>>> serialized after every call to reduce a value into it, always. could this
>>> be the reason it is so slow?
>>>
>>> On Thu, Jan 19, 2017 at 4:17 PM, Koert Kuipers <ko...@tresata.com>
>>> wrote:
>>>
>>>> we just converted a job from RDD to Dataset. the job does a single
>>>> map-red phase using aggregators. we are seeing very bad performance for the
>>>> Dataset version, about 10x slower.
>>>>
>>>> in the Dataset version we use kryo encoders for some of the
>>>> aggregators. based on some basic profiling of spark in local mode i believe
>>>> the bad performance is due to the kryo encoders. about 70% of time is spend
>>>> in kryo related classes.
>>>>
>>>> since we also use kryo for serialization with the RDD i am surprised
>>>> how big the performance difference is.
>>>>
>>>> has anyone seen the same thing? any suggestions for how to improve this?
>>>>
>>>>
>>>
>>
>
Re: dataset aggregators with kryo encoder very slow
Posted by Koert Kuipers <ko...@tresata.com>.
found it :) SPARK-1890
thanks cloud-fan
On Sat, Jan 21, 2017 at 1:46 AM, Koert Kuipers <ko...@tresata.com> wrote:
> trying to replicate this in spark itself i can for v2.1.0 but not for
> master. i guess it has been fixed
>
> On Fri, Jan 20, 2017 at 4:57 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> i started printing out when kryo serializes my buffer data structure for
>> my aggregator.
>>
>> i would expect every buffer object to ideally get serialized only once:
>> at the end of the map-side before the shuffle (so after all the values for
>> the given key within the partition have been reduced into it). i realize
>> that in reality due to the order of the elements coming in this can not
>> always be achieved. but what i see instead is that the buffer is getting
>> serialized after every call to reduce a value into it, always. could this
>> be the reason it is so slow?
>>
>> On Thu, Jan 19, 2017 at 4:17 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> we just converted a job from RDD to Dataset. the job does a single
>>> map-red phase using aggregators. we are seeing very bad performance for the
>>> Dataset version, about 10x slower.
>>>
>>> in the Dataset version we use kryo encoders for some of the aggregators.
>>> based on some basic profiling of spark in local mode i believe the bad
>>> performance is due to the kryo encoders. about 70% of time is spend in kryo
>>> related classes.
>>>
>>> since we also use kryo for serialization with the RDD i am surprised how
>>> big the performance difference is.
>>>
>>> has anyone seen the same thing? any suggestions for how to improve this?
>>>
>>>
>>
>
Re: dataset aggregators with kryo encoder very slow
Posted by Koert Kuipers <ko...@tresata.com>.
trying to replicate this in spark itself i can for v2.1.0 but not for
master. i guess it has been fixed
On Fri, Jan 20, 2017 at 4:57 PM, Koert Kuipers <ko...@tresata.com> wrote:
> i started printing out when kryo serializes my buffer data structure for
> my aggregator.
>
> i would expect every buffer object to ideally get serialized only once: at
> the end of the map-side before the shuffle (so after all the values for the
> given key within the partition have been reduced into it). i realize that
> in reality due to the order of the elements coming in this can not always
> be achieved. but what i see instead is that the buffer is getting
> serialized after every call to reduce a value into it, always. could this
> be the reason it is so slow?
>
> On Thu, Jan 19, 2017 at 4:17 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> we just converted a job from RDD to Dataset. the job does a single
>> map-red phase using aggregators. we are seeing very bad performance for the
>> Dataset version, about 10x slower.
>>
>> in the Dataset version we use kryo encoders for some of the aggregators.
>> based on some basic profiling of spark in local mode i believe the bad
>> performance is due to the kryo encoders. about 70% of time is spend in kryo
>> related classes.
>>
>> since we also use kryo for serialization with the RDD i am surprised how
>> big the performance difference is.
>>
>> has anyone seen the same thing? any suggestions for how to improve this?
>>
>>
>
Re: dataset aggregators with kryo encoder very slow
Posted by Koert Kuipers <ko...@tresata.com>.
i started printing out when kryo serializes my buffer data structure for my
aggregator.
i would expect every buffer object to ideally get serialized only once: at
the end of the map-side before the shuffle (so after all the values for the
given key within the partition have been reduced into it). i realize that
in reality due to the order of the elements coming in this can not always
be achieved. but what i see instead is that the buffer is getting
serialized after every call to reduce a value into it, always. could this
be the reason it is so slow?
On Thu, Jan 19, 2017 at 4:17 PM, Koert Kuipers <ko...@tresata.com> wrote:
> we just converted a job from RDD to Dataset. the job does a single map-red
> phase using aggregators. we are seeing very bad performance for the Dataset
> version, about 10x slower.
>
> in the Dataset version we use kryo encoders for some of the aggregators.
> based on some basic profiling of spark in local mode i believe the bad
> performance is due to the kryo encoders. about 70% of time is spend in kryo
> related classes.
>
> since we also use kryo for serialization with the RDD i am surprised how
> big the performance difference is.
>
> has anyone seen the same thing? any suggestions for how to improve this?
>
>