You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Lasantha Fernando <la...@gmail.com> on 2014/05/13 07:43:20 UTC
Storm Scaling Issues
Hi all,
Is there any guide or hints on how to configure storm to scale better?
I was running some tests with a custom scheduler and found that the
throughput did not scale as expected. Any pointers on what I am doing wrong?
Parallelism24816Single Node (Avg)166099161539.5193986N/ATwo Node (Avg)160988
165563174675.5177624.5
The topology is as follows.
Spout (Generates events continuously) -> Processing Bolt -> Throughput
Measurement Bolt
Parallelism is varied for the processing bolt.
Parallelism for spout and throughput measuring bolt is kept constant at 20
and 1 respectively.
Topology.NUM_WORKERS = 3
Custom scheduler code is available at [1]. Topology code is available at
[2]. Any pointers would be much appreciated.
Thanks,
Lasantha
[1]
https://github.com/sajithshn/storm-schedulers/blob/master/src/main/java/org/wso2/siddhi/storm/scheduler/RoundRobinStormScheduler.java
[2]
https://github.com/lasanthafdo/siddhi-storm/blob/master/src/main/java/org/wso2/siddhi/storm/StockDataTopology.java
Re: Storm Scaling Issues
Posted by Bert Corderman <be...@gmail.com>.
how many ackers? make sure this is set to number of workers.
what do you have for max spout pending? we configured to have 5-10
seconds of messages in pending state. (divide 5-10 seconds by avg tuple
processing time)
Bert
On Wed, May 14, 2014 at 4:35 PM, Lasantha Fernando
<la...@gmail.com>wrote:
> Hi Nathan,
>
> Tried increasing the number of workers with no luck. Also increased the
> parallelism of the throughput measuring bolt and checked the aggregate
> throughput, but that does not seem to be the bottleneck. Also the capacity
> value of the throughput measuring bolt in Storm UI is at around ~0.12.
>
> Will try out more configurations and see. Thank you very much for your
> tips.
>
> Any other tweaks I might try out?
>
> Thanks,
> Lasantha
>
>
> On Tue, May 13, 2014 at 6:38 PM, Nathan Leung <nc...@gmail.com> wrote:
>
>> For 20 spouts and even number of processing bolts 3 seems like an odd
>> number of workers. Also are you sure you're not bottlenecked by your
>> throughput measuring bolt?
>> On May 13, 2014 2:43 AM, "Lasantha Fernando" <la...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> Is there any guide or hints on how to configure storm to scale better?
>>>
>>> I was running some tests with a custom scheduler and found that the
>>> throughput did not scale as expected. Any pointers on what I am doing wrong?
>>>
>>> Parallelism 2 4 8 16 Single Node (Avg) 166099 161539.5 193986 N/A Two
>>> Node (Avg) 160988 165563 174675.5 177624.5
>>> The topology is as follows.
>>>
>>> Spout (Generates events continuously) -> Processing Bolt -> Throughput
>>> Measurement Bolt
>>>
>>> Parallelism is varied for the processing bolt.
>>>
>>> Parallelism for spout and throughput measuring bolt is kept constant at
>>> 20 and 1 respectively.
>>>
>>> Topology.NUM_WORKERS = 3
>>>
>>> Custom scheduler code is available at [1]. Topology code is available at
>>> [2]. Any pointers would be much appreciated.
>>>
>>> Thanks,
>>> Lasantha
>>>
>>> [1]
>>> https://github.com/sajithshn/storm-schedulers/blob/master/src/main/java/org/wso2/siddhi/storm/scheduler/RoundRobinStormScheduler.java
>>> [2]
>>> https://github.com/lasanthafdo/siddhi-storm/blob/master/src/main/java/org/wso2/siddhi/storm/StockDataTopology.java
>>>
>>
>
unsubscribe
Posted by eric perler <er...@hotmail.com>.
unsubscribe
Re: Storm Scaling Issues
Posted by Nathan Leung <nc...@gmail.com>.
Hi,
Is your logging set to debug? If you get a lot of log messages that can
slow you down. How fast is your network? Do you need fields grouping on
your bolt? (it looks like yes but can be worthwhile to re-evaluate whether
this is the case). You can try the following:
1) run with just spouts, check throughput. Judging by your spout this
should be very high
2) run with your spout, and the processing bolt. Comment the business
logic of your bolt to reduce the variables. Check the throughput with
fieldsGrouping, shuffleGrouping, and localOrShuffleGrouping. If
localOrShuffleGrouping is significantly faster than the other two, then you
might be running into so kind of networking bottleneck. Judging by your
numbers in the original email I suspect this to be the case.
3) Add the throughput measuring bolt and check performance through your
entire flow.
-Nathan
On Wed, May 14, 2014 at 4:35 PM, Lasantha Fernando
<la...@gmail.com>wrote:
> Hi Nathan,
>
> Tried increasing the number of workers with no luck. Also increased the
> parallelism of the throughput measuring bolt and checked the aggregate
> throughput, but that does not seem to be the bottleneck. Also the capacity
> value of the throughput measuring bolt in Storm UI is at around ~0.12.
>
> Will try out more configurations and see. Thank you very much for your
> tips.
>
> Any other tweaks I might try out?
>
> Thanks,
> Lasantha
>
>
> On Tue, May 13, 2014 at 6:38 PM, Nathan Leung <nc...@gmail.com> wrote:
>
>> For 20 spouts and even number of processing bolts 3 seems like an odd
>> number of workers. Also are you sure you're not bottlenecked by your
>> throughput measuring bolt?
>> On May 13, 2014 2:43 AM, "Lasantha Fernando" <la...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> Is there any guide or hints on how to configure storm to scale better?
>>>
>>> I was running some tests with a custom scheduler and found that the
>>> throughput did not scale as expected. Any pointers on what I am doing wrong?
>>>
>>> Parallelism 2 4 8 16 Single Node (Avg) 166099 161539.5 193986 N/A Two
>>> Node (Avg) 160988 165563 174675.5 177624.5
>>> The topology is as follows.
>>>
>>> Spout (Generates events continuously) -> Processing Bolt -> Throughput
>>> Measurement Bolt
>>>
>>> Parallelism is varied for the processing bolt.
>>>
>>> Parallelism for spout and throughput measuring bolt is kept constant at
>>> 20 and 1 respectively.
>>>
>>> Topology.NUM_WORKERS = 3
>>>
>>> Custom scheduler code is available at [1]. Topology code is available at
>>> [2]. Any pointers would be much appreciated.
>>>
>>> Thanks,
>>> Lasantha
>>>
>>> [1]
>>> https://github.com/sajithshn/storm-schedulers/blob/master/src/main/java/org/wso2/siddhi/storm/scheduler/RoundRobinStormScheduler.java
>>> [2]
>>> https://github.com/lasanthafdo/siddhi-storm/blob/master/src/main/java/org/wso2/siddhi/storm/StockDataTopology.java
>>>
>>
>
Re: Storm Scaling Issues
Posted by Lasantha Fernando <la...@gmail.com>.
Hi Nathan,
Tried increasing the number of workers with no luck. Also increased the
parallelism of the throughput measuring bolt and checked the aggregate
throughput, but that does not seem to be the bottleneck. Also the capacity
value of the throughput measuring bolt in Storm UI is at around ~0.12.
Will try out more configurations and see. Thank you very much for your tips.
Any other tweaks I might try out?
Thanks,
Lasantha
On Tue, May 13, 2014 at 6:38 PM, Nathan Leung <nc...@gmail.com> wrote:
> For 20 spouts and even number of processing bolts 3 seems like an odd
> number of workers. Also are you sure you're not bottlenecked by your
> throughput measuring bolt?
> On May 13, 2014 2:43 AM, "Lasantha Fernando" <la...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> Is there any guide or hints on how to configure storm to scale better?
>>
>> I was running some tests with a custom scheduler and found that the
>> throughput did not scale as expected. Any pointers on what I am doing wrong?
>>
>> Parallelism 2 4 8 16 Single Node (Avg) 166099 161539.5 193986 N/A Two
>> Node (Avg) 160988 165563 174675.5 177624.5
>> The topology is as follows.
>>
>> Spout (Generates events continuously) -> Processing Bolt -> Throughput
>> Measurement Bolt
>>
>> Parallelism is varied for the processing bolt.
>>
>> Parallelism for spout and throughput measuring bolt is kept constant at
>> 20 and 1 respectively.
>>
>> Topology.NUM_WORKERS = 3
>>
>> Custom scheduler code is available at [1]. Topology code is available at
>> [2]. Any pointers would be much appreciated.
>>
>> Thanks,
>> Lasantha
>>
>> [1]
>> https://github.com/sajithshn/storm-schedulers/blob/master/src/main/java/org/wso2/siddhi/storm/scheduler/RoundRobinStormScheduler.java
>> [2]
>> https://github.com/lasanthafdo/siddhi-storm/blob/master/src/main/java/org/wso2/siddhi/storm/StockDataTopology.java
>>
>
Re: Storm Scaling Issues
Posted by Nathan Leung <nc...@gmail.com>.
For 20 spouts and even number of processing bolts 3 seems like an odd
number of workers. Also are you sure you're not bottlenecked by your
throughput measuring bolt?
On May 13, 2014 2:43 AM, "Lasantha Fernando" <la...@gmail.com> wrote:
> Hi all,
>
> Is there any guide or hints on how to configure storm to scale better?
>
> I was running some tests with a custom scheduler and found that the
> throughput did not scale as expected. Any pointers on what I am doing wrong?
>
> Parallelism 2 4 8 16 Single Node (Avg) 166099 161539.5 193986 N/A Two
> Node (Avg) 160988 165563 174675.5 177624.5
> The topology is as follows.
>
> Spout (Generates events continuously) -> Processing Bolt -> Throughput
> Measurement Bolt
>
> Parallelism is varied for the processing bolt.
>
> Parallelism for spout and throughput measuring bolt is kept constant at 20
> and 1 respectively.
>
> Topology.NUM_WORKERS = 3
>
> Custom scheduler code is available at [1]. Topology code is available at
> [2]. Any pointers would be much appreciated.
>
> Thanks,
> Lasantha
>
> [1]
> https://github.com/sajithshn/storm-schedulers/blob/master/src/main/java/org/wso2/siddhi/storm/scheduler/RoundRobinStormScheduler.java
> [2]
> https://github.com/lasanthafdo/siddhi-storm/blob/master/src/main/java/org/wso2/siddhi/storm/StockDataTopology.java
>