You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Manpreet Singh <ma...@sourcefuse.com> on 2015/01/28 19:59:34 UTC
auto scaling apache storm
Hi,
I am a newbie to Apache Storm and analyzing it for a product. I am
comparing it with Gearman from parallel processing aspect. So far Apache
Storm seems to be winning on most of the aspects. But one of the important
aspect which I got stuck is 'auto-scale'. Auto-scale (up as well as down)
of storm resources doesn't look possible. This is critical for my product
as stream data could vary a lot through out the day. Having auto-scale will
save me cost as well as monitoring the usage every now and then.
Are there any workarounds to this problem with Apache Storm?
Thanks in advance.
-Manpreet.
Re: auto scaling apache storm
Posted by Ahmed El Rheddane <ah...@imag.fr>.
I believe rebalancing should be triggered by the user, using the
command "storm rebalance" or its API equivalent. The rest is about right :)
Good luck,
Ahmed
On 01/30/2015 03:46 PM, Manpreet Singh wrote:
> Thanks a lot Ahmed.
>
> It really gave me a direction. So you are saying that:
>
> * A program should check metrics at regular intervals.
> * Based on that, program can add/remove VMs to cluster. Storm will
> automatically recognize the change in VMs
> * Depending on VM addition or removal, program will trigger
> re-balance command to change no. of worker processes or executors.
>
> Did I get it right?
>
> Thanks,
> Manpreet.
>
> On Thu, Jan 29, 2015 at 2:50 PM, Ahmed El Rheddane
> <ahmed.el-rheddane@imag.fr <ma...@imag.fr>> wrote:
>
> Hi all,
>
> I have investigated ways of auto scaling Storm topologies recently
> and it is perfectly feasible.
> - For monitoring, you can use either the Thrift interface or the
> new MetricsConsumer bolt, it's probably possible using the REST
> API as well, but I haven't tried that out yet.
> - For scaling, Storm automatically discovers the addition of new
> nodes (VMs) to the cluster and allows you to change the number of
> executors (threads) per component on the fly.
> - You can as well plug in a custom scheduler, which is called upon
> each rebalancing request, to place the executors of your topology
> as you see fits.
>
> I hope this could help.
>
> Ahmed
>
>
> On 01/28/2015 09:55 PM, Parth Brahmbhatt wrote:
>
> Not sure how much progress has been made on
> https://issues.apache.org/jira/browse/STORM-594 but that is
> the only
> initiative that I am aware of.
>
> Thanks
> Parth
>
> On 1/28/15, 12:42 PM, "Margus Roo" <margus@roo.ee
> <ma...@roo.ee>> wrote:
>
> I have not heard or read about it from the documentation.
>
> Margus (margusja) Roo
> http://margus.roo.ee
> skype: margusja
> +372 51 480
>
> On 28/01/15 20:59, Manpreet Singh wrote:
>
> Hi,
>
> I am a newbie to Apache Storm and analyzing it for a
> product. I am
> comparing it with Gearman from parallel processing
> aspect. So far
> Apache Storm seems to be winning on most of the
> aspects. But one of
> the important aspect which I got stuck is
> 'auto-scale'. Auto-scale (up
> as well as down) of storm resources doesn't look
> possible. This is
> critical for my product as stream data could vary a
> lot through out
> the day. Having auto-scale will save me cost as well
> as monitoring the
> usage every now and then.
>
> Are there any workarounds to this problem with Apache
> Storm?
>
> Thanks in advance.
>
> -Manpreet.
>
>
>
Re: auto scaling apache storm
Posted by Manpreet Singh <ma...@sourcefuse.com>.
Thanks a lot Ahmed.
It really gave me a direction. So you are saying that:
- A program should check metrics at regular intervals.
- Based on that, program can add/remove VMs to cluster. Storm will
automatically recognize the change in VMs
- Depending on VM addition or removal, program will trigger re-balance
command to change no. of worker processes or executors.
Did I get it right?
Thanks,
Manpreet.
On Thu, Jan 29, 2015 at 2:50 PM, Ahmed El Rheddane <
ahmed.el-rheddane@imag.fr> wrote:
> Hi all,
>
> I have investigated ways of auto scaling Storm topologies recently and it
> is perfectly feasible.
> - For monitoring, you can use either the Thrift interface or the new
> MetricsConsumer bolt, it's probably possible using the REST API as well,
> but I haven't tried that out yet.
> - For scaling, Storm automatically discovers the addition of new nodes
> (VMs) to the cluster and allows you to change the number of executors
> (threads) per component on the fly.
> - You can as well plug in a custom scheduler, which is called upon each
> rebalancing request, to place the executors of your topology as you see
> fits.
>
> I hope this could help.
>
> Ahmed
>
>
> On 01/28/2015 09:55 PM, Parth Brahmbhatt wrote:
>
>> Not sure how much progress has been made on
>> https://issues.apache.org/jira/browse/STORM-594 but that is the only
>> initiative that I am aware of.
>>
>> Thanks
>> Parth
>>
>> On 1/28/15, 12:42 PM, "Margus Roo" <ma...@roo.ee> wrote:
>>
>> I have not heard or read about it from the documentation.
>>>
>>> Margus (margusja) Roo
>>> http://margus.roo.ee
>>> skype: margusja
>>> +372 51 480
>>>
>>> On 28/01/15 20:59, Manpreet Singh wrote:
>>>
>>>> Hi,
>>>>
>>>> I am a newbie to Apache Storm and analyzing it for a product. I am
>>>> comparing it with Gearman from parallel processing aspect. So far
>>>> Apache Storm seems to be winning on most of the aspects. But one of
>>>> the important aspect which I got stuck is 'auto-scale'. Auto-scale (up
>>>> as well as down) of storm resources doesn't look possible. This is
>>>> critical for my product as stream data could vary a lot through out
>>>> the day. Having auto-scale will save me cost as well as monitoring the
>>>> usage every now and then.
>>>>
>>>> Are there any workarounds to this problem with Apache Storm?
>>>>
>>>> Thanks in advance.
>>>>
>>>> -Manpreet.
>>>>
>>>
>
Re: auto scaling apache storm
Posted by Ahmed El Rheddane <ah...@imag.fr>.
Hi all,
I have investigated ways of auto scaling Storm topologies recently and
it is perfectly feasible.
- For monitoring, you can use either the Thrift interface or the new
MetricsConsumer bolt, it's probably possible using the REST API as well,
but I haven't tried that out yet.
- For scaling, Storm automatically discovers the addition of new nodes
(VMs) to the cluster and allows you to change the number of executors
(threads) per component on the fly.
- You can as well plug in a custom scheduler, which is called upon each
rebalancing request, to place the executors of your topology as you see
fits.
I hope this could help.
Ahmed
On 01/28/2015 09:55 PM, Parth Brahmbhatt wrote:
> Not sure how much progress has been made on
> https://issues.apache.org/jira/browse/STORM-594 but that is the only
> initiative that I am aware of.
>
> Thanks
> Parth
>
> On 1/28/15, 12:42 PM, "Margus Roo" <ma...@roo.ee> wrote:
>
>> I have not heard or read about it from the documentation.
>>
>> Margus (margusja) Roo
>> http://margus.roo.ee
>> skype: margusja
>> +372 51 480
>>
>> On 28/01/15 20:59, Manpreet Singh wrote:
>>> Hi,
>>>
>>> I am a newbie to Apache Storm and analyzing it for a product. I am
>>> comparing it with Gearman from parallel processing aspect. So far
>>> Apache Storm seems to be winning on most of the aspects. But one of
>>> the important aspect which I got stuck is 'auto-scale'. Auto-scale (up
>>> as well as down) of storm resources doesn't look possible. This is
>>> critical for my product as stream data could vary a lot through out
>>> the day. Having auto-scale will save me cost as well as monitoring the
>>> usage every now and then.
>>>
>>> Are there any workarounds to this problem with Apache Storm?
>>>
>>> Thanks in advance.
>>>
>>> -Manpreet.
Re: auto scaling apache storm
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
Not sure how much progress has been made on
https://issues.apache.org/jira/browse/STORM-594 but that is the only
initiative that I am aware of.
Thanks
Parth
On 1/28/15, 12:42 PM, "Margus Roo" <ma...@roo.ee> wrote:
>I have not heard or read about it from the documentation.
>
>Margus (margusja) Roo
>http://margus.roo.ee
>skype: margusja
>+372 51 480
>
>On 28/01/15 20:59, Manpreet Singh wrote:
>> Hi,
>>
>> I am a newbie to Apache Storm and analyzing it for a product. I am
>> comparing it with Gearman from parallel processing aspect. So far
>> Apache Storm seems to be winning on most of the aspects. But one of
>> the important aspect which I got stuck is 'auto-scale'. Auto-scale (up
>> as well as down) of storm resources doesn't look possible. This is
>> critical for my product as stream data could vary a lot through out
>> the day. Having auto-scale will save me cost as well as monitoring the
>> usage every now and then.
>>
>> Are there any workarounds to this problem with Apache Storm?
>>
>> Thanks in advance.
>>
>> -Manpreet.
>
Re: auto scaling apache storm
Posted by Margus Roo <ma...@roo.ee>.
I have not heard or read about it from the documentation.
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 480
On 28/01/15 20:59, Manpreet Singh wrote:
> Hi,
>
> I am a newbie to Apache Storm and analyzing it for a product. I am
> comparing it with Gearman from parallel processing aspect. So far
> Apache Storm seems to be winning on most of the aspects. But one of
> the important aspect which I got stuck is 'auto-scale'. Auto-scale (up
> as well as down) of storm resources doesn't look possible. This is
> critical for my product as stream data could vary a lot through out
> the day. Having auto-scale will save me cost as well as monitoring the
> usage every now and then.
>
> Are there any workarounds to this problem with Apache Storm?
>
> Thanks in advance.
>
> -Manpreet.