You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Anandh Kumar <an...@gmail.com> on 2016/04/16 16:01:37 UTC
Storm 0.10.0 Benchmark issue
Hi Team,
I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not getting
throughput more than 7000 per seconds.
Right now getting following throughput.
Ack thread: 4
max.spout.pending: 250
With KafkaSpout with simple bolt
Throughput - 7000/s message
======================================================
Ack thread: 4
max.spout.pending: 250
With KafkaSpout with* no bolt *
Throughput - 100000/s message
====================================================
Why with bolt I am getting very less throughput?
Not getting thoughput more than 7000/s. Tried to tune all the buffer
parameters( send,receive buffer) not use.
Please help me.
Regards,
-Anandh Kumar
Re: Storm 0.10.0 Benchmark issue
Posted by Anandh Kumar <an...@gmail.com>.
Hi Nikos,
My topic have 6 partition so my kafka-spouts parallelism hint also 6.
My worker is 10
KafkaSpout executor is 6
Only with Kafka-spout without bolt I got 1,00,000/s but excepted 10,00,000/s
Regards,
-Anandh Kumar
On Mon, Apr 18, 2016 at 8:53 PM, Nikos R. Katsipoulakis <
nick.katsip@gmail.com> wrote:
> Hello again,
>
> I am not using kafka-spouts and I did not run any benchmarks on my setup.
> Can I ask what is your parallelism hint on the kafka-spouts? Because, it
> might be the case that 7000 t/s might be the ceiling for your parallelism
> hint and you may have to increase that in order to achieve higher
> throughput.
>
> Cheers,
> Nikos
>
> On Mon, Apr 18, 2016 at 1:56 AM, Anandh Kumar <an...@gmail.com>
> wrote:
>
>> Nikos,
>>
>> Thanks for your reply.
>>
>> I tried many value from low to high of max.spout.pending but not getting
>> throughput more than 7000/s.
>>
>> I tried turning internal buffer but no improvement.
>>
>> Did you tried any benchmark test on storm kafkaspout ? how much you got?
>> given the code so that I can run same test in my environment.
>>
>> Regards,
>> -Anandh Kumar
>>
>> On Sat, Apr 16, 2016 at 8:44 PM, Nikos R. Katsipoulakis <
>> nick.katsip@gmail.com> wrote:
>>
>>> Hello Anandh,
>>>
>>> Why do you keep the max.spout.pending this low? I suggest increasing it
>>> to a higher value, because this might be the cause of your low throughput.
>>> In fact, it makes sense if you consider the following scenario:
>>>
>>> The KafkaSpout sends out 250 messages that try to fill out the Netty
>>> buffers before they are sent to the next node. When those are send, the
>>> Spout stops sending any additional tuples, until the ones "on-flight" are
>>> fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time
>>> is spent up until they are sent out. Therefore, your throughput is
>>> implicitly "throttled"
>>>
>>> If I were you, I would keep the default values on the buffers (or even
>>> increase them a little - common practice for high throughput) and also try
>>> with a bigger value on max.spout.pending.
>>>
>>> Cheers,
>>> Nikos
>>>
>>> On Sat, Apr 16, 2016 at 11:06 AM, John Yost <ho...@gmail.com>
>>> wrote:
>>>
>>>> Hi Anandh,
>>>>
>>>> I recommend profiling your workers with something like jvisualvm to see
>>>> where your Bolt executors are spending their time.
>>>>
>>>> --John
>>>>
>>>> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not
>>>>> getting throughput more than 7000 per seconds.
>>>>>
>>>>> Right now getting following throughput.
>>>>>
>>>>> Ack thread: 4
>>>>> max.spout.pending: 250
>>>>>
>>>>> With KafkaSpout with simple bolt
>>>>>
>>>>> Throughput - 7000/s message
>>>>>
>>>>> ======================================================
>>>>>
>>>>> Ack thread: 4
>>>>> max.spout.pending: 250
>>>>>
>>>>> With KafkaSpout with* no bolt *
>>>>>
>>>>> Throughput - 100000/s message
>>>>>
>>>>> ====================================================
>>>>>
>>>>> Why with bolt I am getting very less throughput?
>>>>>
>>>>> Not getting thoughput more than 7000/s. Tried to tune all the buffer
>>>>> parameters( send,receive buffer) not use.
>>>>>
>>>>> Please help me.
>>>>>
>>>>> Regards,
>>>>> -Anandh Kumar
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Nikos R. Katsipoulakis,
>>> Department of Computer Science
>>> University of Pittsburgh
>>>
>>
>>
>
>
> --
> Nikos R. Katsipoulakis,
> Department of Computer Science
> University of Pittsburgh
>
Re: Storm 0.10.0 Benchmark issue
Posted by "Nikos R. Katsipoulakis" <ni...@gmail.com>.
Hello again,
I am not using kafka-spouts and I did not run any benchmarks on my setup.
Can I ask what is your parallelism hint on the kafka-spouts? Because, it
might be the case that 7000 t/s might be the ceiling for your parallelism
hint and you may have to increase that in order to achieve higher
throughput.
Cheers,
Nikos
On Mon, Apr 18, 2016 at 1:56 AM, Anandh Kumar <an...@gmail.com> wrote:
> Nikos,
>
> Thanks for your reply.
>
> I tried many value from low to high of max.spout.pending but not getting
> throughput more than 7000/s.
>
> I tried turning internal buffer but no improvement.
>
> Did you tried any benchmark test on storm kafkaspout ? how much you got?
> given the code so that I can run same test in my environment.
>
> Regards,
> -Anandh Kumar
>
> On Sat, Apr 16, 2016 at 8:44 PM, Nikos R. Katsipoulakis <
> nick.katsip@gmail.com> wrote:
>
>> Hello Anandh,
>>
>> Why do you keep the max.spout.pending this low? I suggest increasing it
>> to a higher value, because this might be the cause of your low throughput.
>> In fact, it makes sense if you consider the following scenario:
>>
>> The KafkaSpout sends out 250 messages that try to fill out the Netty
>> buffers before they are sent to the next node. When those are send, the
>> Spout stops sending any additional tuples, until the ones "on-flight" are
>> fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time
>> is spent up until they are sent out. Therefore, your throughput is
>> implicitly "throttled"
>>
>> If I were you, I would keep the default values on the buffers (or even
>> increase them a little - common practice for high throughput) and also try
>> with a bigger value on max.spout.pending.
>>
>> Cheers,
>> Nikos
>>
>> On Sat, Apr 16, 2016 at 11:06 AM, John Yost <ho...@gmail.com> wrote:
>>
>>> Hi Anandh,
>>>
>>> I recommend profiling your workers with something like jvisualvm to see
>>> where your Bolt executors are spending their time.
>>>
>>> --John
>>>
>>> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <an...@gmail.com>
>>> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not
>>>> getting throughput more than 7000 per seconds.
>>>>
>>>> Right now getting following throughput.
>>>>
>>>> Ack thread: 4
>>>> max.spout.pending: 250
>>>>
>>>> With KafkaSpout with simple bolt
>>>>
>>>> Throughput - 7000/s message
>>>>
>>>> ======================================================
>>>>
>>>> Ack thread: 4
>>>> max.spout.pending: 250
>>>>
>>>> With KafkaSpout with* no bolt *
>>>>
>>>> Throughput - 100000/s message
>>>>
>>>> ====================================================
>>>>
>>>> Why with bolt I am getting very less throughput?
>>>>
>>>> Not getting thoughput more than 7000/s. Tried to tune all the buffer
>>>> parameters( send,receive buffer) not use.
>>>>
>>>> Please help me.
>>>>
>>>> Regards,
>>>> -Anandh Kumar
>>>>
>>>
>>>
>>
>>
>> --
>> Nikos R. Katsipoulakis,
>> Department of Computer Science
>> University of Pittsburgh
>>
>
>
--
Nikos R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh
Re: Storm 0.10.0 Benchmark issue
Posted by Anandh Kumar <an...@gmail.com>.
Nikos,
Thanks for your reply.
I tried many value from low to high of max.spout.pending but not getting
throughput more than 7000/s.
I tried turning internal buffer but no improvement.
Did you tried any benchmark test on storm kafkaspout ? how much you got?
given the code so that I can run same test in my environment.
Regards,
-Anandh Kumar
On Sat, Apr 16, 2016 at 8:44 PM, Nikos R. Katsipoulakis <
nick.katsip@gmail.com> wrote:
> Hello Anandh,
>
> Why do you keep the max.spout.pending this low? I suggest increasing it to
> a higher value, because this might be the cause of your low throughput. In
> fact, it makes sense if you consider the following scenario:
>
> The KafkaSpout sends out 250 messages that try to fill out the Netty
> buffers before they are sent to the next node. When those are send, the
> Spout stops sending any additional tuples, until the ones "on-flight" are
> fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time
> is spent up until they are sent out. Therefore, your throughput is
> implicitly "throttled"
>
> If I were you, I would keep the default values on the buffers (or even
> increase them a little - common practice for high throughput) and also try
> with a bigger value on max.spout.pending.
>
> Cheers,
> Nikos
>
> On Sat, Apr 16, 2016 at 11:06 AM, John Yost <ho...@gmail.com> wrote:
>
>> Hi Anandh,
>>
>> I recommend profiling your workers with something like jvisualvm to see
>> where your Bolt executors are spending their time.
>>
>> --John
>>
>> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <an...@gmail.com>
>> wrote:
>>
>>> Hi Team,
>>>
>>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not getting
>>> throughput more than 7000 per seconds.
>>>
>>> Right now getting following throughput.
>>>
>>> Ack thread: 4
>>> max.spout.pending: 250
>>>
>>> With KafkaSpout with simple bolt
>>>
>>> Throughput - 7000/s message
>>>
>>> ======================================================
>>>
>>> Ack thread: 4
>>> max.spout.pending: 250
>>>
>>> With KafkaSpout with* no bolt *
>>>
>>> Throughput - 100000/s message
>>>
>>> ====================================================
>>>
>>> Why with bolt I am getting very less throughput?
>>>
>>> Not getting thoughput more than 7000/s. Tried to tune all the buffer
>>> parameters( send,receive buffer) not use.
>>>
>>> Please help me.
>>>
>>> Regards,
>>> -Anandh Kumar
>>>
>>
>>
>
>
> --
> Nikos R. Katsipoulakis,
> Department of Computer Science
> University of Pittsburgh
>
Re: Storm 0.10.0 Benchmark issue
Posted by "Nikos R. Katsipoulakis" <ni...@gmail.com>.
Hello Anandh,
Why do you keep the max.spout.pending this low? I suggest increasing it to
a higher value, because this might be the cause of your low throughput. In
fact, it makes sense if you consider the following scenario:
The KafkaSpout sends out 250 messages that try to fill out the Netty
buffers before they are sent to the next node. When those are send, the
Spout stops sending any additional tuples, until the ones "on-flight" are
fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time
is spent up until they are sent out. Therefore, your throughput is
implicitly "throttled"
If I were you, I would keep the default values on the buffers (or even
increase them a little - common practice for high throughput) and also try
with a bigger value on max.spout.pending.
Cheers,
Nikos
On Sat, Apr 16, 2016 at 11:06 AM, John Yost <ho...@gmail.com> wrote:
> Hi Anandh,
>
> I recommend profiling your workers with something like jvisualvm to see
> where your Bolt executors are spending their time.
>
> --John
>
> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <an...@gmail.com>
> wrote:
>
>> Hi Team,
>>
>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not getting
>> throughput more than 7000 per seconds.
>>
>> Right now getting following throughput.
>>
>> Ack thread: 4
>> max.spout.pending: 250
>>
>> With KafkaSpout with simple bolt
>>
>> Throughput - 7000/s message
>>
>> ======================================================
>>
>> Ack thread: 4
>> max.spout.pending: 250
>>
>> With KafkaSpout with* no bolt *
>>
>> Throughput - 100000/s message
>>
>> ====================================================
>>
>> Why with bolt I am getting very less throughput?
>>
>> Not getting thoughput more than 7000/s. Tried to tune all the buffer
>> parameters( send,receive buffer) not use.
>>
>> Please help me.
>>
>> Regards,
>> -Anandh Kumar
>>
>
>
--
Nikos R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh
Re: Storm 0.10.0 Benchmark issue
Posted by John Yost <ho...@gmail.com>.
Hi Anandh,
I recommend profiling your workers with something like jvisualvm to see
where your Bolt executors are spending their time.
--John
On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <an...@gmail.com> wrote:
> Hi Team,
>
> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not getting
> throughput more than 7000 per seconds.
>
> Right now getting following throughput.
>
> Ack thread: 4
> max.spout.pending: 250
>
> With KafkaSpout with simple bolt
>
> Throughput - 7000/s message
>
> ======================================================
>
> Ack thread: 4
> max.spout.pending: 250
>
> With KafkaSpout with* no bolt *
>
> Throughput - 100000/s message
>
> ====================================================
>
> Why with bolt I am getting very less throughput?
>
> Not getting thoughput more than 7000/s. Tried to tune all the buffer
> parameters( send,receive buffer) not use.
>
> Please help me.
>
> Regards,
> -Anandh Kumar
>