How to stress test storm?

[Amila Suriarachchi <am...@gmail.com>]

hi,

I am a student at CSU and currently I am doing a performance comparison
among a stream processing system I am developing[1], s4 and storm.

As I have written here[2], we can stress test s4 by using number of threads
(eg. 100) at an adapter and sending them through a set of processing
elements. My problem is how can I write a similar test with storm?

Storm has the same concepts spouts and bolts. In a spout it polls messages
using its' own thread. In storm we can set a parameter for parallelism.
However as I understood this is the number of spout instances I need to run
within in the cluster and always one spout instance invoke with one thread.
Is there a way to make one spout invoke from 100 threads?

S4 has a concept of cluster which we can use to control where we want to
deploy our processing elements and adapters. Is it possible to do a similar
thing in storm where I can specify which nodes I want to run spouts and
bolts?

thanks,
Amila.


[1] https://github.com/amilaSuriarachchi/stream
[2]
https://github.com/amilaSuriarachchi/s4-samples/tree/master/src/main/java/edu/colostate/cs/count


-- 
Amila Suriarachchi
blog: http://amilachinthaka.blogspot.com/

Re: How to stress test storm?

[Amila Suriarachchi <am...@gmail.com>]

Thanks for your information. I'll go through your links and see.

Amila.


On Mon, Aug 18, 2014 at 11:52 AM, Spico Florin <sp...@gmail.com>
wrote:

> Hello!
>   The spout or bolt are running as instances (tasks) inside in an executor
> threa. You can run more tasks per executor thread by they will tun
> sequential.  You can read more about the storm parallelism here:
>
>
> http://storm.incubator.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
>
> Regarding your question about how to specify where to place the spout and
> bolts on nodes what you need is a scheduler class that will be put in a
> library that will be added to nimbus lib folder. You can find more
> information here
>
> http://storm.incubator.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
>
> and here
>
> https://github.com/xumingming/storm-lib/blob/master/src/jvm/storm/DemoScheduler.java
>
> I have  done someting similar for my project and it worked.
> I hope that these help.
>  Regards,
>  Florin
>
>
>
>
> On Sun, Aug 17, 2014 at 6:29 PM, Amila Suriarachchi <
> amilasuriarachchi@gmail.com> wrote:
>
>> hi,
>>
>> I am a student at CSU and currently I am doing a performance comparison
>> among a stream processing system I am developing[1], s4 and storm.
>>
>> As I have written here[2], we can stress test s4 by using number of
>> threads (eg. 100) at an adapter and sending them through a set of
>> processing elements. My problem is how can I write a similar test with
>> storm?
>>
>> Storm has the same concepts spouts and bolts. In a spout it polls
>> messages using its' own thread. In storm we can set a parameter for
>> parallelism. However as I understood this is the number of spout instances
>> I need to run within in the cluster and always one spout instance invoke
>> with one thread. Is there a way to make one spout invoke from 100 threads?
>>
>> S4 has a concept of cluster which we can use to control where we want to
>> deploy our processing elements and adapters. Is it possible to do a similar
>> thing in storm where I can specify which nodes I want to run spouts and
>> bolts?
>>
>> thanks,
>> Amila.
>>
>>
>> [1] https://github.com/amilaSuriarachchi/stream
>> [2]
>> https://github.com/amilaSuriarachchi/s4-samples/tree/master/src/main/java/edu/colostate/cs/count
>>
>>
>> --
>> Amila Suriarachchi
>> blog: http://amilachinthaka.blogspot.com/
>>
>
>


-- 
Amila Suriarachchi
blog: http://amilachinthaka.blogspot.com/

Re: How to stress test storm?

[Spico Florin <sp...@gmail.com>]

Hello!
  The spout or bolt are running as instances (tasks) inside in an executor
threa. You can run more tasks per executor thread by they will tun
sequential.  You can read more about the storm parallelism here:


http://storm.incubator.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html

Regarding your question about how to specify where to place the spout and
bolts on nodes what you need is a scheduler class that will be put in a
library that will be added to nimbus lib folder. You can find more
information here
http://storm.incubator.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html

and here
https://github.com/xumingming/storm-lib/blob/master/src/jvm/storm/DemoScheduler.java

I have  done someting similar for my project and it worked.
I hope that these help.
 Regards,
 Florin




On Sun, Aug 17, 2014 at 6:29 PM, Amila Suriarachchi <
amilasuriarachchi@gmail.com> wrote:

> hi,
>
> I am a student at CSU and currently I am doing a performance comparison
> among a stream processing system I am developing[1], s4 and storm.
>
> As I have written here[2], we can stress test s4 by using number of
> threads (eg. 100) at an adapter and sending them through a set of
> processing elements. My problem is how can I write a similar test with
> storm?
>
> Storm has the same concepts spouts and bolts. In a spout it polls messages
> using its' own thread. In storm we can set a parameter for parallelism.
> However as I understood this is the number of spout instances I need to run
> within in the cluster and always one spout instance invoke with one thread.
> Is there a way to make one spout invoke from 100 threads?
>
> S4 has a concept of cluster which we can use to control where we want to
> deploy our processing elements and adapters. Is it possible to do a similar
> thing in storm where I can specify which nodes I want to run spouts and
> bolts?
>
> thanks,
> Amila.
>
>
> [1] https://github.com/amilaSuriarachchi/stream
> [2]
> https://github.com/amilaSuriarachchi/s4-samples/tree/master/src/main/java/edu/colostate/cs/count
>
>
> --
> Amila Suriarachchi
> blog: http://amilachinthaka.blogspot.com/
>