You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by Andronidis Anastasios <an...@hotmail.com> on 2014/09/19 16:52:58 UTC
MapWritable is really bad
Hello,
I remember a discussion rose upon performance issues on messages and that kryo serializer helped a lot.
Please read this: http://www.chrisstucchio.com/blog/2011/mapwritable_sometimes_a_performance_hog.html
From a custom test I did, I was sending some messages with MapWritable as a container, Text as key and ArrayWritable (with integers inside) as a value. Hama was reporting 20MB of traffic. When I wrote my own Map (that implements Writable interface) I reduced the amount from 20MB to 1.5MB..
Cheers,
Anastasios
Re: MapWritable is really bad
Posted by Chia-Hung Lin <cl...@googlemail.com>.
I mean in addition to kyro, we also roll our own implementation for
mapwriable as it seems (IIRC) we are going to keep both (kyro and
writable) in our hama framework.
Sorry for not articulating clearly.
On 25 September 2014 19:18, Andronidis Anastasios
<an...@hotmail.com> wrote:
> Hi, I am not saying to remove kryo. Kryo is a wonderful asset to have. I just say that kryo + careful data structure use = the best performance.
>
> We must make sure that kryo is running optimally though. The library has lots of configurations and we must make sure that we use it properly.
>
> Cheers,
> Anastasis
>
> On 25 Σεπ 2014, at 12:58 μ.μ., Chia-Hung Lin <cl...@googlemail.com> wrote:
>
>> Should we roll out with our own implementation? Switching to kyro look
>> like will get of this issue, but if we are going to have plugable
>> serialization framework (are we?) that might help those who need it.
>>
>> On 24 September 2014 17:19, Edward J. Yoon <ed...@apache.org> wrote:
>>> Interesting ..
>>>
>>> On Fri, Sep 19, 2014 at 11:52 PM, Andronidis Anastasios
>>> <an...@hotmail.com> wrote:
>>>> Hello,
>>>>
>>>> I remember a discussion rose upon performance issues on messages and that kryo serializer helped a lot.
>>>>
>>>> Please read this: http://www.chrisstucchio.com/blog/2011/mapwritable_sometimes_a_performance_hog.html
>>>>
>>>> From a custom test I did, I was sending some messages with MapWritable as a container, Text as key and ArrayWritable (with integers inside) as a value. Hama was reporting 20MB of traffic. When I wrote my own Map (that implements Writable interface) I reduced the amount from 20MB to 1.5MB..
>>>>
>>>> Cheers,
>>>> Anastasios
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> CEO at DataSayer Co., Ltd.
>
Re: MapWritable is really bad
Posted by Andronidis Anastasios <an...@hotmail.com>.
Hi, I am not saying to remove kryo. Kryo is a wonderful asset to have. I just say that kryo + careful data structure use = the best performance.
We must make sure that kryo is running optimally though. The library has lots of configurations and we must make sure that we use it properly.
Cheers,
Anastasis
On 25 Σεπ 2014, at 12:58 μ.μ., Chia-Hung Lin <cl...@googlemail.com> wrote:
> Should we roll out with our own implementation? Switching to kyro look
> like will get of this issue, but if we are going to have plugable
> serialization framework (are we?) that might help those who need it.
>
> On 24 September 2014 17:19, Edward J. Yoon <ed...@apache.org> wrote:
>> Interesting ..
>>
>> On Fri, Sep 19, 2014 at 11:52 PM, Andronidis Anastasios
>> <an...@hotmail.com> wrote:
>>> Hello,
>>>
>>> I remember a discussion rose upon performance issues on messages and that kryo serializer helped a lot.
>>>
>>> Please read this: http://www.chrisstucchio.com/blog/2011/mapwritable_sometimes_a_performance_hog.html
>>>
>>> From a custom test I did, I was sending some messages with MapWritable as a container, Text as key and ArrayWritable (with integers inside) as a value. Hama was reporting 20MB of traffic. When I wrote my own Map (that implements Writable interface) I reduced the amount from 20MB to 1.5MB..
>>>
>>> Cheers,
>>> Anastasios
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> CEO at DataSayer Co., Ltd.
Re: MapWritable is really bad
Posted by Chia-Hung Lin <cl...@googlemail.com>.
Should we roll out with our own implementation? Switching to kyro look
like will get of this issue, but if we are going to have plugable
serialization framework (are we?) that might help those who need it.
On 24 September 2014 17:19, Edward J. Yoon <ed...@apache.org> wrote:
> Interesting ..
>
> On Fri, Sep 19, 2014 at 11:52 PM, Andronidis Anastasios
> <an...@hotmail.com> wrote:
>> Hello,
>>
>> I remember a discussion rose upon performance issues on messages and that kryo serializer helped a lot.
>>
>> Please read this: http://www.chrisstucchio.com/blog/2011/mapwritable_sometimes_a_performance_hog.html
>>
>> From a custom test I did, I was sending some messages with MapWritable as a container, Text as key and ArrayWritable (with integers inside) as a value. Hama was reporting 20MB of traffic. When I wrote my own Map (that implements Writable interface) I reduced the amount from 20MB to 1.5MB..
>>
>> Cheers,
>> Anastasios
>
>
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.
Re: MapWritable is really bad
Posted by "Edward J. Yoon" <ed...@apache.org>.
Interesting ..
On Fri, Sep 19, 2014 at 11:52 PM, Andronidis Anastasios
<an...@hotmail.com> wrote:
> Hello,
>
> I remember a discussion rose upon performance issues on messages and that kryo serializer helped a lot.
>
> Please read this: http://www.chrisstucchio.com/blog/2011/mapwritable_sometimes_a_performance_hog.html
>
> From a custom test I did, I was sending some messages with MapWritable as a container, Text as key and ArrayWritable (with integers inside) as a value. Hama was reporting 20MB of traffic. When I wrote my own Map (that implements Writable interface) I reduced the amount from 20MB to 1.5MB..
>
> Cheers,
> Anastasios
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.