You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Klausen Schaefersinho <kl...@gmail.com> on 2014/01/10 14:40:12 UTC
Storm Performance
Hi,
how is the performance of a real cluster compared to a local development
cluster? I was trying to benchmark Storm now and found that it only
manages to consume 10 events per second in a simple topology which only
consists out of i spout and one bolt where the spout creates random objects
and the bolt acknowledge the tuple.
10 events terrible slow, and I am not sure if this is related to the
develop mode.
Cheers,
Klaus
Re[2]: Storm Performance
Posted by ch...@gmail.com.
About the performance of Storm, I have done a performance test. With write/read hbase and redis, my topology could still process about 3000 tuples per second in our StormCluster server.
Sent from myMail for iOS
2014年1月10日 星期五 23:54 +0800 from Michael Ritsema <mi...@gmail.com>:
I deploy with LocalCluster mode. I get several thousands messages processed a second but expect my botteneck is not at the storm layer.
It would be interesting to see a full explanation of the differences in LocalCluster and a real deployment. I expect quite a few people would benefit from having storm run in process in actual deployments.
-Rits
On Fri, Jan 10, 2014 at 9:46 AM, Michael Rose < michael@fullcontact.com > wrote:
>Post your code. Even Dev mode is far faster for us.
>On Jan 10, 2014 8:44 AM, "Klausen Schaefersinho" < klaus.schaefers@gmail.com > wrote:
>>> I've benched storm at 1.8 million tuples per second on a big (24 core) box using local or shuffle grouping between a spout and bolt.
>>Production or development mode?
>>
>>> you're only seeing 10 events per second make sure you don't have any sleeps Yeah I checked for sleeps etc. and stripped down my code. Now my bolts do nothing and the spout just creates random data...
>>
>>
>>
>>
>>On Fri, Jan 10, 2014 at 4:37 PM, Nathan Leung < ncleung@gmail.com > wrote:
>>>I've benched storm at 1.8 million tuples per second on a big (24 core) box using local or shuffle grouping between a spout and bolt. If you're only seeing 10 events per second make sure you don't have any sleeps (whether in your code or elsewhere e.g. a library or triggered due to lack of data in the storm spout). Also note that the default sleep period in the spout when there is no data (in storm 0.9) is I believe 1ms so even if you're hitting this condition you should see much more than 10 events per second.
>>>On Jan 10, 2014 10:19 AM, "Jon Logan" < jmlogan@buffalo.edu > wrote:
>>>>Why would you benchmark local mode? It's intended for debugging and development purposes, not actual production use...............
>>>>
>>>>
>>>>per the website, Storm itself is benchmarked at > 1 million tuples per second per node.
>>>>
>>>>
>>>>On Fri, Jan 10, 2014 at 8:40 AM, Klausen Schaefersinho < klaus.schaefers@gmail.com > wrote:
>>>>>Hi,
>>>>>
>>>>>how is the performance of a real cluster compared to a local development cluster? I was trying to benchmark Storm now and found that it only manages to consume 10 events per second in a simple topology which only consists out of i spout and one bolt where the spout creates random objects and the bolt acknowledge the tuple.
>>>>>
>>>>>10 events terrible slow, and I am not sure if this is related to the develop mode.
>>>>>
>>>>>Cheers,
>>>>>
>>>>>Klaus
>>
Re: Storm Performance
Posted by Michael Ritsema <mi...@gmail.com>.
I deploy with LocalCluster mode. I get several thousands messages processed
a second but expect my botteneck is not at the storm layer.
It would be interesting to see a full explanation of the differences in
LocalCluster and a real deployment. I expect quite a few people would
benefit from having storm run in process in actual deployments.
-Rits
On Fri, Jan 10, 2014 at 9:46 AM, Michael Rose <mi...@fullcontact.com>wrote:
> Post your code. Even Dev mode is far faster for us.
> On Jan 10, 2014 8:44 AM, "Klausen Schaefersinho" <
> klaus.schaefers@gmail.com> wrote:
>
>> > I've benched storm at 1.8 million tuples per second on a big (24 core)
>> box using local or shuffle grouping between a spout and bolt.
>> Production or development mode?
>>
>> > you're only seeing 10 events per second make sure you don't have any
>> sleeps
>> Yeah I checked for sleeps etc. and stripped down my code. Now my bolts
>> do nothing and the spout just creates random data...
>>
>>
>>
>>
>> On Fri, Jan 10, 2014 at 4:37 PM, Nathan Leung <nc...@gmail.com> wrote:
>>
>>> I've benched storm at 1.8 million tuples per second on a big (24 core)
>>> box using local or shuffle grouping between a spout and bolt. If you're
>>> only seeing 10 events per second make sure you don't have any sleeps
>>> (whether in your code or elsewhere e.g. a library or triggered due to lack
>>> of data in the storm spout). Also note that the default sleep period in the
>>> spout when there is no data (in storm 0.9) is I believe 1ms so even if
>>> you're hitting this condition you should see much more than 10 events per
>>> second.
>>> On Jan 10, 2014 10:19 AM, "Jon Logan" <jm...@buffalo.edu> wrote:
>>>
>>>> Why would you benchmark local mode? It's intended for debugging and
>>>> development purposes, not actual production use...............
>>>>
>>>>
>>>> per the website, Storm itself is benchmarked at > 1 million tuples per
>>>> second per node.
>>>>
>>>>
>>>> On Fri, Jan 10, 2014 at 8:40 AM, Klausen Schaefersinho <
>>>> klaus.schaefers@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> how is the performance of a real cluster compared to a local
>>>>> development cluster? I was trying to benchmark Storm now and found that it
>>>>> only manages to consume 10 events per second in a simple topology which
>>>>> only consists out of i spout and one bolt where the spout creates random
>>>>> objects and the bolt acknowledge the tuple.
>>>>>
>>>>> 10 events terrible slow, and I am not sure if this is related to the
>>>>> develop mode.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Klaus
>>>>>
>>>>
>>>>
>>
Re: Storm Performance
Posted by Michael Rose <mi...@fullcontact.com>.
Post your code. Even Dev mode is far faster for us.
On Jan 10, 2014 8:44 AM, "Klausen Schaefersinho" <kl...@gmail.com>
wrote:
> > I've benched storm at 1.8 million tuples per second on a big (24 core)
> box using local or shuffle grouping between a spout and bolt.
> Production or development mode?
>
> > you're only seeing 10 events per second make sure you don't have any
> sleeps
> Yeah I checked for sleeps etc. and stripped down my code. Now my bolts do
> nothing and the spout just creates random data...
>
>
>
>
> On Fri, Jan 10, 2014 at 4:37 PM, Nathan Leung <nc...@gmail.com> wrote:
>
>> I've benched storm at 1.8 million tuples per second on a big (24 core)
>> box using local or shuffle grouping between a spout and bolt. If you're
>> only seeing 10 events per second make sure you don't have any sleeps
>> (whether in your code or elsewhere e.g. a library or triggered due to lack
>> of data in the storm spout). Also note that the default sleep period in the
>> spout when there is no data (in storm 0.9) is I believe 1ms so even if
>> you're hitting this condition you should see much more than 10 events per
>> second.
>> On Jan 10, 2014 10:19 AM, "Jon Logan" <jm...@buffalo.edu> wrote:
>>
>>> Why would you benchmark local mode? It's intended for debugging and
>>> development purposes, not actual production use...............
>>>
>>>
>>> per the website, Storm itself is benchmarked at > 1 million tuples per
>>> second per node.
>>>
>>>
>>> On Fri, Jan 10, 2014 at 8:40 AM, Klausen Schaefersinho <
>>> klaus.schaefers@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> how is the performance of a real cluster compared to a local
>>>> development cluster? I was trying to benchmark Storm now and found that it
>>>> only manages to consume 10 events per second in a simple topology which
>>>> only consists out of i spout and one bolt where the spout creates random
>>>> objects and the bolt acknowledge the tuple.
>>>>
>>>> 10 events terrible slow, and I am not sure if this is related to the
>>>> develop mode.
>>>>
>>>> Cheers,
>>>>
>>>> Klaus
>>>>
>>>
>>>
>
Re: Storm Performance
Posted by Klausen Schaefersinho <kl...@gmail.com>.
> I've benched storm at 1.8 million tuples per second on a big (24 core)
box using local or shuffle grouping between a spout and bolt.
Production or development mode?
> you're only seeing 10 events per second make sure you don't have any
sleeps
Yeah I checked for sleeps etc. and stripped down my code. Now my bolts do
nothing and the spout just creates random data...
On Fri, Jan 10, 2014 at 4:37 PM, Nathan Leung <nc...@gmail.com> wrote:
> I've benched storm at 1.8 million tuples per second on a big (24 core) box
> using local or shuffle grouping between a spout and bolt. If you're only
> seeing 10 events per second make sure you don't have any sleeps (whether in
> your code or elsewhere e.g. a library or triggered due to lack of data in
> the storm spout). Also note that the default sleep period in the spout when
> there is no data (in storm 0.9) is I believe 1ms so even if you're hitting
> this condition you should see much more than 10 events per second.
> On Jan 10, 2014 10:19 AM, "Jon Logan" <jm...@buffalo.edu> wrote:
>
>> Why would you benchmark local mode? It's intended for debugging and
>> development purposes, not actual production use...............
>>
>>
>> per the website, Storm itself is benchmarked at > 1 million tuples per
>> second per node.
>>
>>
>> On Fri, Jan 10, 2014 at 8:40 AM, Klausen Schaefersinho <
>> klaus.schaefers@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> how is the performance of a real cluster compared to a local development
>>> cluster? I was trying to benchmark Storm now and found that it only
>>> manages to consume 10 events per second in a simple topology which only
>>> consists out of i spout and one bolt where the spout creates random objects
>>> and the bolt acknowledge the tuple.
>>>
>>> 10 events terrible slow, and I am not sure if this is related to the
>>> develop mode.
>>>
>>> Cheers,
>>>
>>> Klaus
>>>
>>
>>
Re: Storm Performance
Posted by Nathan Leung <nc...@gmail.com>.
I've benched storm at 1.8 million tuples per second on a big (24 core) box
using local or shuffle grouping between a spout and bolt. If you're only
seeing 10 events per second make sure you don't have any sleeps (whether in
your code or elsewhere e.g. a library or triggered due to lack of data in
the storm spout). Also note that the default sleep period in the spout when
there is no data (in storm 0.9) is I believe 1ms so even if you're hitting
this condition you should see much more than 10 events per second.
On Jan 10, 2014 10:19 AM, "Jon Logan" <jm...@buffalo.edu> wrote:
> Why would you benchmark local mode? It's intended for debugging and
> development purposes, not actual production use...............
>
>
> per the website, Storm itself is benchmarked at > 1 million tuples per
> second per node.
>
>
> On Fri, Jan 10, 2014 at 8:40 AM, Klausen Schaefersinho <
> klaus.schaefers@gmail.com> wrote:
>
>> Hi,
>>
>> how is the performance of a real cluster compared to a local development
>> cluster? I was trying to benchmark Storm now and found that it only
>> manages to consume 10 events per second in a simple topology which only
>> consists out of i spout and one bolt where the spout creates random objects
>> and the bolt acknowledge the tuple.
>>
>> 10 events terrible slow, and I am not sure if this is related to the
>> develop mode.
>>
>> Cheers,
>>
>> Klaus
>>
>
>
Re: Storm Performance
Posted by Klausen Schaefersinho <kl...@gmail.com>.
Hi,
I am know evaluation some technologies, like classic JMS, Samza etc.
Therefore I was bench-marking in local mode, as it was the fastest way to
do it. Also I would have expected that it should be close to a production
system running on one node, as you don't have network overhead etc...
>Storm itself is benchmarked at > 1 million tuples per second per node.
Yes, I remember seeing that, but I didn't see any comments with regards to
the hardware the used.
Cheers,
Klaus
On Fri, Jan 10, 2014 at 4:17 PM, Jon Logan <jm...@buffalo.edu> wrote:
> Why would you benchmark local mode? It's intended for debugging and
> development purposes, not actual production use...............
>
>
> per the website, Storm itself is benchmarked at > 1 million tuples per
> second per node.
>
>
> On Fri, Jan 10, 2014 at 8:40 AM, Klausen Schaefersinho <
> klaus.schaefers@gmail.com> wrote:
>
>> Hi,
>>
>> how is the performance of a real cluster compared to a local development
>> cluster? I was trying to benchmark Storm now and found that it only
>> manages to consume 10 events per second in a simple topology which only
>> consists out of i spout and one bolt where the spout creates random objects
>> and the bolt acknowledge the tuple.
>>
>> 10 events terrible slow, and I am not sure if this is related to the
>> develop mode.
>>
>> Cheers,
>>
>> Klaus
>>
>
>
Re: Storm Performance
Posted by Jon Logan <jm...@buffalo.edu>.
Why would you benchmark local mode? It's intended for debugging and
development purposes, not actual production use...............
per the website, Storm itself is benchmarked at > 1 million tuples per
second per node.
On Fri, Jan 10, 2014 at 8:40 AM, Klausen Schaefersinho <
klaus.schaefers@gmail.com> wrote:
> Hi,
>
> how is the performance of a real cluster compared to a local development
> cluster? I was trying to benchmark Storm now and found that it only
> manages to consume 10 events per second in a simple topology which only
> consists out of i spout and one bolt where the spout creates random objects
> and the bolt acknowledge the tuple.
>
> 10 events terrible slow, and I am not sure if this is related to the
> develop mode.
>
> Cheers,
>
> Klaus
>