You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Dmitry Goldenberg <dg...@gmail.com> on 2015/06/02 17:31:42 UTC

Cluster autoscaling in Storm?

Hi,

What's the latest perspective on cluster autoscaling in Storm? We want to
be able to have machines provisioned/added or decommissioned via AWS based
on the system load.

E.g. this post:
https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ
didn't seem to get an answer.

Is there a way to pre-provision a pool of machines ready to go but not
fully in use, then use them as needed?

I see
https://issues.apache.org/jira/browse/STORM-594
marked as Minor - ?

This writeup seems to indicate there's autoscaling in Storm:
https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3C54CB9BA8.4070009@imag.fr%3E

Any thoughts/recommendations? Thanks.

Re: Cluster autoscaling in Storm?

Posted by Dillian Murphey <cr...@gmail.com>.
Never mind, I didn't look closely enough at the docs  The rebalance options
are "optional" and the /0 at the end is the seconds to wait before it does
the rebalance. :P


On Tue, Jun 30, 2015 at 5:44 PM, Dillian Murphey <cr...@gmail.com>
wrote:

> Actually, can someone explain the replace REST API.
>
> curl  -i -b ~/cookiejar.txt -c ~/cookiejar.txt -X POST
> -H "Content-Type: application/json"
> -d  '{"rebalanceOptions": {"numWorkers": 2, "executors": { "spout" : "5", "split": 7, "count": 5 }}, "callback":"foo"}' http://localhost:8080/api/v1/topology/wordcount-1-1420308665/rebalance/0
>
> I take it /0 is a reference to a topology?  How do I know what values to
> use for the executors and numworkers?
>
> Through the Storm UI, it's just a button. Anyone know how it determines
> these values?
>
> Thanks
>
>
> On Tue, Jun 30, 2015 at 5:26 PM, Dillian Murphey <cr...@gmail.com>
> wrote:
>
>> Anyone have any updates on automatically re-balancing storm workers when
>> using an autoscaling group?
>>
>> On Mon, Jun 22, 2015 at 2:50 AM, Andrew Wu <aw...@innometrics.com> wrote:
>>
>>>  I agree with you.  I think there are 2 levels of scaling issue
>>> 1) Topology level, mainly on number of workers. we are setting 2 twice
>>> amount of workers than number of supervisors so that even we double number
>>> of supervisors there will be at least 1 vm on each supervisor. it works on
>>> sharing cpu load but does not work on memory.
>>> 2) Cluster level, storm will not automatically use new supervisor until
>>> topology is "rebalanced”.
>>>
>>>  I have investigated this a bit and here’s something may help
>>> This can be a solution for 2)
>>> https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md. you
>>> can have a saperate monitor program to detect cluster changes and rebalance
>>> topologies programatically.
>>> 1) is a bit hard to solve, I*d like to know if there is any solution out
>>> there as well.
>>>
>>>
>>>
>>>   *Andrew Ge Wu*
>>>  Backend Developer & Integration Tech Lead
>>> Mobile: +46 76 88 99 970
>>>
>>>  Website: www.innometrics.com
>>>
>>>
>>>
>>>  On 02 Jun 2015, at 17:31, Dmitry Goldenberg <dg...@gmail.com>
>>> wrote:
>>>
>>>  Hi,
>>>
>>>  What's the latest perspective on cluster autoscaling in Storm? We want
>>> to be able to have machines provisioned/added or decommissioned via AWS
>>> based on the system load.
>>>
>>>  E.g. this post:
>>> https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ
>>> didn't seem to get an answer.
>>>
>>>  Is there a way to pre-provision a pool of machines ready to go but not
>>> fully in use, then use them as needed?
>>>
>>>  I see
>>>  https://issues.apache.org/jira/browse/STORM-594
>>> marked as Minor - ?
>>>
>>>  This writeup seems to indicate there's autoscaling in Storm:
>>>
>>> https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3C54CB9BA8.4070009@imag.fr%3E
>>>
>>>  Any thoughts/recommendations? Thanks.
>>>
>>>
>>>
>>
>

Re: Cluster autoscaling in Storm?

Posted by Dillian Murphey <cr...@gmail.com>.
Actually, can someone explain the replace REST API.

curl  -i -b ~/cookiejar.txt -c ~/cookiejar.txt -X POST
-H "Content-Type: application/json"
-d  '{"rebalanceOptions": {"numWorkers": 2, "executors": { "spout" :
"5", "split": 7, "count": 5 }}, "callback":"foo"}'
http://localhost:8080/api/v1/topology/wordcount-1-1420308665/rebalance/0

I take it /0 is a reference to a topology?  How do I know what values to
use for the executors and numworkers?

Through the Storm UI, it's just a button. Anyone know how it determines
these values?

Thanks


On Tue, Jun 30, 2015 at 5:26 PM, Dillian Murphey <cr...@gmail.com>
wrote:

> Anyone have any updates on automatically re-balancing storm workers when
> using an autoscaling group?
>
> On Mon, Jun 22, 2015 at 2:50 AM, Andrew Wu <aw...@innometrics.com> wrote:
>
>>  I agree with you.  I think there are 2 levels of scaling issue
>> 1) Topology level, mainly on number of workers. we are setting 2 twice
>> amount of workers than number of supervisors so that even we double number
>> of supervisors there will be at least 1 vm on each supervisor. it works on
>> sharing cpu load but does not work on memory.
>> 2) Cluster level, storm will not automatically use new supervisor until
>> topology is "rebalanced”.
>>
>>  I have investigated this a bit and here’s something may help
>> This can be a solution for 2)
>> https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md. you
>> can have a saperate monitor program to detect cluster changes and rebalance
>> topologies programatically.
>> 1) is a bit hard to solve, I*d like to know if there is any solution out
>> there as well.
>>
>>
>>
>>   *Andrew Ge Wu*
>>  Backend Developer & Integration Tech Lead
>> Mobile: +46 76 88 99 970
>>
>>  Website: www.innometrics.com
>>
>>
>>
>>  On 02 Jun 2015, at 17:31, Dmitry Goldenberg <dg...@gmail.com>
>> wrote:
>>
>>  Hi,
>>
>>  What's the latest perspective on cluster autoscaling in Storm? We want
>> to be able to have machines provisioned/added or decommissioned via AWS
>> based on the system load.
>>
>>  E.g. this post:
>> https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ
>> didn't seem to get an answer.
>>
>>  Is there a way to pre-provision a pool of machines ready to go but not
>> fully in use, then use them as needed?
>>
>>  I see
>>  https://issues.apache.org/jira/browse/STORM-594
>> marked as Minor - ?
>>
>>  This writeup seems to indicate there's autoscaling in Storm:
>>
>> https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3C54CB9BA8.4070009@imag.fr%3E
>>
>>  Any thoughts/recommendations? Thanks.
>>
>>
>>
>

Re: Cluster autoscaling in Storm?

Posted by Dillian Murphey <cr...@gmail.com>.
Anyone have any updates on automatically re-balancing storm workers when
using an autoscaling group?

On Mon, Jun 22, 2015 at 2:50 AM, Andrew Wu <aw...@innometrics.com> wrote:

>  I agree with you.  I think there are 2 levels of scaling issue
> 1) Topology level, mainly on number of workers. we are setting 2 twice
> amount of workers than number of supervisors so that even we double number
> of supervisors there will be at least 1 vm on each supervisor. it works on
> sharing cpu load but does not work on memory.
> 2) Cluster level, storm will not automatically use new supervisor until
> topology is "rebalanced”.
>
>  I have investigated this a bit and here’s something may help
> This can be a solution for 2)
> https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md. you can
> have a saperate monitor program to detect cluster changes and rebalance
> topologies programatically.
> 1) is a bit hard to solve, I*d like to know if there is any solution out
> there as well.
>
>
>
>   *Andrew Ge Wu*
>  Backend Developer & Integration Tech Lead
> Mobile: +46 76 88 99 970
>
>  Website: www.innometrics.com
>
>
>
>  On 02 Jun 2015, at 17:31, Dmitry Goldenberg <dg...@gmail.com>
> wrote:
>
>  Hi,
>
>  What's the latest perspective on cluster autoscaling in Storm? We want
> to be able to have machines provisioned/added or decommissioned via AWS
> based on the system load.
>
>  E.g. this post:
> https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ
> didn't seem to get an answer.
>
>  Is there a way to pre-provision a pool of machines ready to go but not
> fully in use, then use them as needed?
>
>  I see
>  https://issues.apache.org/jira/browse/STORM-594
> marked as Minor - ?
>
>  This writeup seems to indicate there's autoscaling in Storm:
>
> https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3C54CB9BA8.4070009@imag.fr%3E
>
>  Any thoughts/recommendations? Thanks.
>
>
>

Re: Cluster autoscaling in Storm?

Posted by Andrew Wu <aw...@innometrics.com>.
I agree with you.  I think there are 2 levels of scaling issue
1) Topology level, mainly on number of workers. we are setting 2 twice amount of workers than number of supervisors so that even we double number of supervisors there will be at least 1 vm on each supervisor. it works on sharing cpu load but does not work on memory.
2) Cluster level, storm will not automatically use new supervisor until topology is "rebalanced”.

I have investigated this a bit and here’s something may help
This can be a solution for 2) https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md. you can have a saperate monitor program to detect cluster changes and rebalance topologies programatically.
1) is a bit hard to solve, I*d like to know if there is any solution out there as well.



Andrew Ge Wu
Backend Developer & Integration Tech Lead
Mobile: +46 76 88 99 970
Website: www.innometrics.com<http://www.innometrics.com/>


[cid:A29501B8-EA20-463B-A4A4-203BB3DA4788]

On 02 Jun 2015, at 17:31, Dmitry Goldenberg <dg...@gmail.com>> wrote:

Hi,

What's the latest perspective on cluster autoscaling in Storm? We want to be able to have machines provisioned/added or decommissioned via AWS based on the system load.

E.g. this post: https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ didn't seem to get an answer.

Is there a way to pre-provision a pool of machines ready to go but not fully in use, then use them as needed?

I see
https://issues.apache.org/jira/browse/STORM-594
marked as Minor - ?

This writeup seems to indicate there's autoscaling in Storm:
https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3C54CB9BA8.4070009@imag.fr%3E

Any thoughts/recommendations? Thanks.