You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Dmitry Goldenberg <dg...@gmail.com> on 2015/06/02 17:31:42 UTC
Cluster autoscaling in Storm?
Hi,
What's the latest perspective on cluster autoscaling in Storm? We want to
be able to have machines provisioned/added or decommissioned via AWS based
on the system load.
E.g. this post:
https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ
didn't seem to get an answer.
Is there a way to pre-provision a pool of machines ready to go but not
fully in use, then use them as needed?
I see
https://issues.apache.org/jira/browse/STORM-594
marked as Minor - ?
This writeup seems to indicate there's autoscaling in Storm:
https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3C54CB9BA8.4070009@imag.fr%3E
Any thoughts/recommendations? Thanks.
Re: Cluster autoscaling in Storm?
Posted by Dillian Murphey <cr...@gmail.com>.
Never mind, I didn't look closely enough at the docs The rebalance options
are "optional" and the /0 at the end is the seconds to wait before it does
the rebalance. :P
On Tue, Jun 30, 2015 at 5:44 PM, Dillian Murphey <cr...@gmail.com>
wrote:
> Actually, can someone explain the replace REST API.
>
> curl -i -b ~/cookiejar.txt -c ~/cookiejar.txt -X POST
> -H "Content-Type: application/json"
> -d '{"rebalanceOptions": {"numWorkers": 2, "executors": { "spout" : "5", "split": 7, "count": 5 }}, "callback":"foo"}' http://localhost:8080/api/v1/topology/wordcount-1-1420308665/rebalance/0
>
> I take it /0 is a reference to a topology? How do I know what values to
> use for the executors and numworkers?
>
> Through the Storm UI, it's just a button. Anyone know how it determines
> these values?
>
> Thanks
>
>
> On Tue, Jun 30, 2015 at 5:26 PM, Dillian Murphey <cr...@gmail.com>
> wrote:
>
>> Anyone have any updates on automatically re-balancing storm workers when
>> using an autoscaling group?
>>
>> On Mon, Jun 22, 2015 at 2:50 AM, Andrew Wu <aw...@innometrics.com> wrote:
>>
>>> I agree with you. I think there are 2 levels of scaling issue
>>> 1) Topology level, mainly on number of workers. we are setting 2 twice
>>> amount of workers than number of supervisors so that even we double number
>>> of supervisors there will be at least 1 vm on each supervisor. it works on
>>> sharing cpu load but does not work on memory.
>>> 2) Cluster level, storm will not automatically use new supervisor until
>>> topology is "rebalanced”.
>>>
>>> I have investigated this a bit and here’s something may help
>>> This can be a solution for 2)
>>> https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md. you
>>> can have a saperate monitor program to detect cluster changes and rebalance
>>> topologies programatically.
>>> 1) is a bit hard to solve, I*d like to know if there is any solution out
>>> there as well.
>>>
>>>
>>>
>>> *Andrew Ge Wu*
>>> Backend Developer & Integration Tech Lead
>>> Mobile: +46 76 88 99 970
>>>
>>> Website: www.innometrics.com
>>>
>>>
>>>
>>> On 02 Jun 2015, at 17:31, Dmitry Goldenberg <dg...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> What's the latest perspective on cluster autoscaling in Storm? We want
>>> to be able to have machines provisioned/added or decommissioned via AWS
>>> based on the system load.
>>>
>>> E.g. this post:
>>> https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ
>>> didn't seem to get an answer.
>>>
>>> Is there a way to pre-provision a pool of machines ready to go but not
>>> fully in use, then use them as needed?
>>>
>>> I see
>>> https://issues.apache.org/jira/browse/STORM-594
>>> marked as Minor - ?
>>>
>>> This writeup seems to indicate there's autoscaling in Storm:
>>>
>>> https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3C54CB9BA8.4070009@imag.fr%3E
>>>
>>> Any thoughts/recommendations? Thanks.
>>>
>>>
>>>
>>
>
Re: Cluster autoscaling in Storm?
Posted by Dillian Murphey <cr...@gmail.com>.
Actually, can someone explain the replace REST API.
curl -i -b ~/cookiejar.txt -c ~/cookiejar.txt -X POST
-H "Content-Type: application/json"
-d '{"rebalanceOptions": {"numWorkers": 2, "executors": { "spout" :
"5", "split": 7, "count": 5 }}, "callback":"foo"}'
http://localhost:8080/api/v1/topology/wordcount-1-1420308665/rebalance/0
I take it /0 is a reference to a topology? How do I know what values to
use for the executors and numworkers?
Through the Storm UI, it's just a button. Anyone know how it determines
these values?
Thanks
On Tue, Jun 30, 2015 at 5:26 PM, Dillian Murphey <cr...@gmail.com>
wrote:
> Anyone have any updates on automatically re-balancing storm workers when
> using an autoscaling group?
>
> On Mon, Jun 22, 2015 at 2:50 AM, Andrew Wu <aw...@innometrics.com> wrote:
>
>> I agree with you. I think there are 2 levels of scaling issue
>> 1) Topology level, mainly on number of workers. we are setting 2 twice
>> amount of workers than number of supervisors so that even we double number
>> of supervisors there will be at least 1 vm on each supervisor. it works on
>> sharing cpu load but does not work on memory.
>> 2) Cluster level, storm will not automatically use new supervisor until
>> topology is "rebalanced”.
>>
>> I have investigated this a bit and here’s something may help
>> This can be a solution for 2)
>> https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md. you
>> can have a saperate monitor program to detect cluster changes and rebalance
>> topologies programatically.
>> 1) is a bit hard to solve, I*d like to know if there is any solution out
>> there as well.
>>
>>
>>
>> *Andrew Ge Wu*
>> Backend Developer & Integration Tech Lead
>> Mobile: +46 76 88 99 970
>>
>> Website: www.innometrics.com
>>
>>
>>
>> On 02 Jun 2015, at 17:31, Dmitry Goldenberg <dg...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> What's the latest perspective on cluster autoscaling in Storm? We want
>> to be able to have machines provisioned/added or decommissioned via AWS
>> based on the system load.
>>
>> E.g. this post:
>> https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ
>> didn't seem to get an answer.
>>
>> Is there a way to pre-provision a pool of machines ready to go but not
>> fully in use, then use them as needed?
>>
>> I see
>> https://issues.apache.org/jira/browse/STORM-594
>> marked as Minor - ?
>>
>> This writeup seems to indicate there's autoscaling in Storm:
>>
>> https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3C54CB9BA8.4070009@imag.fr%3E
>>
>> Any thoughts/recommendations? Thanks.
>>
>>
>>
>
Re: Cluster autoscaling in Storm?
Posted by Dillian Murphey <cr...@gmail.com>.
Anyone have any updates on automatically re-balancing storm workers when
using an autoscaling group?
On Mon, Jun 22, 2015 at 2:50 AM, Andrew Wu <aw...@innometrics.com> wrote:
> I agree with you. I think there are 2 levels of scaling issue
> 1) Topology level, mainly on number of workers. we are setting 2 twice
> amount of workers than number of supervisors so that even we double number
> of supervisors there will be at least 1 vm on each supervisor. it works on
> sharing cpu load but does not work on memory.
> 2) Cluster level, storm will not automatically use new supervisor until
> topology is "rebalanced”.
>
> I have investigated this a bit and here’s something may help
> This can be a solution for 2)
> https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md. you can
> have a saperate monitor program to detect cluster changes and rebalance
> topologies programatically.
> 1) is a bit hard to solve, I*d like to know if there is any solution out
> there as well.
>
>
>
> *Andrew Ge Wu*
> Backend Developer & Integration Tech Lead
> Mobile: +46 76 88 99 970
>
> Website: www.innometrics.com
>
>
>
> On 02 Jun 2015, at 17:31, Dmitry Goldenberg <dg...@gmail.com>
> wrote:
>
> Hi,
>
> What's the latest perspective on cluster autoscaling in Storm? We want
> to be able to have machines provisioned/added or decommissioned via AWS
> based on the system load.
>
> E.g. this post:
> https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ
> didn't seem to get an answer.
>
> Is there a way to pre-provision a pool of machines ready to go but not
> fully in use, then use them as needed?
>
> I see
> https://issues.apache.org/jira/browse/STORM-594
> marked as Minor - ?
>
> This writeup seems to indicate there's autoscaling in Storm:
>
> https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3C54CB9BA8.4070009@imag.fr%3E
>
> Any thoughts/recommendations? Thanks.
>
>
>
Re: Cluster autoscaling in Storm?
Posted by Andrew Wu <aw...@innometrics.com>.
I agree with you. I think there are 2 levels of scaling issue
1) Topology level, mainly on number of workers. we are setting 2 twice amount of workers than number of supervisors so that even we double number of supervisors there will be at least 1 vm on each supervisor. it works on sharing cpu load but does not work on memory.
2) Cluster level, storm will not automatically use new supervisor until topology is "rebalanced”.
I have investigated this a bit and here’s something may help
This can be a solution for 2) https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md. you can have a saperate monitor program to detect cluster changes and rebalance topologies programatically.
1) is a bit hard to solve, I*d like to know if there is any solution out there as well.
Andrew Ge Wu
Backend Developer & Integration Tech Lead
Mobile: +46 76 88 99 970
Website: www.innometrics.com<http://www.innometrics.com/>
[cid:A29501B8-EA20-463B-A4A4-203BB3DA4788]
On 02 Jun 2015, at 17:31, Dmitry Goldenberg <dg...@gmail.com>> wrote:
Hi,
What's the latest perspective on cluster autoscaling in Storm? We want to be able to have machines provisioned/added or decommissioned via AWS based on the system load.
E.g. this post: https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ didn't seem to get an answer.
Is there a way to pre-provision a pool of machines ready to go but not fully in use, then use them as needed?
I see
https://issues.apache.org/jira/browse/STORM-594
marked as Minor - ?
This writeup seems to indicate there's autoscaling in Storm:
https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3C54CB9BA8.4070009@imag.fr%3E
Any thoughts/recommendations? Thanks.