You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by hilfi alkaff <al...@illinois.edu> on 2014/03/19 04:42:57 UTC
Storm rebalance details.
Hi,
I would like to know how does Storm currently perform rebalance. I see in
*./storm-core/src/clj/backtype/storm/daemon/nimbus.clj* there's a
*do-rebalance* method that will call *update-storm!* (Which I assume
changes the state of Storm) but I don't see where the method
*update-storm!*is being defined.
Thanks,
--
Hilfi M. Alkaff
Re: Storm rebalance details.
Posted by hilfi alkaff <hi...@gmail.com>.
Thanks, I am currently still going through the code. So the default
scheduler seems to be the EvenScheduler which basically tries to spread out
all the workers to all the available slots correct?
On Fri, Mar 21, 2014 at 8:48 AM, Bobby Evans <ev...@yahoo-inc.com> wrote:
> You are correct. Rebalance updates the state so when mk-assignments is
> called the scheduler does not see any workers associated with that
> topology. It then will proceeded to reschedule it wherever the scheduler
> thinks is best. Storm does not do any dynamic rebalancing yet. But if
> you have a killer algorithm for it I would be happy to help update it so
> that it could.
>
> ―Bobby
>
> On 3/21/14, 1:18 AM, "hilfi alkaff" <hi...@gmail.com> wrote:
>
> >Walk me through if I'm understanding the code correctly. I'm still very
> >new
> >to clojure. In cluster.clj, update-storm! just change the internal state
> >variables, but it has not actually changed the state of the cluster? The
> >actual changes happen at mk-assignments when it calls assignSlots?
> >
> >Additionally, storm does not perform any dynamic rebalancing in the fly
> >correct? Or has there been a plan to do that? Thanks.
> >
> >
> >On Wed, Mar 19, 2014 at 12:18 AM, 杨晓青 <xq...@gmail.com> wrote:
> >
> >> in file /storm-core/src/clj/backtype/storm/cluster.clj
> >>
> >>
> >> 2014-03-19 11:42 GMT+08:00 hilfi alkaff <al...@illinois.edu>:
> >>
> >> > Hi,
> >> >
> >> > I would like to know how does Storm currently perform rebalance. I
> >>see in
> >> > *./storm-core/src/clj/backtype/storm/daemon/nimbus.clj* there's a
> >> > *do-rebalance* method that will call *update-storm!* (Which I assume
> >> > changes the state of Storm) but I don't see where the method
> >> > *update-storm!*is being defined.
> >> >
> >> > Thanks,
> >> >
> >> > --
> >> > Hilfi M. Alkaff
> >> >
> >>
> >>
> >>
> >> --
> >> -------------------------------------------
> >> best regards
> >>
> >> 杨晓青
> >>
> >
> >
> >
> >--
> >~Hilfi Alkaff~
>
>
--
~Hilfi Alkaff~
Re: Storm rebalance details.
Posted by Bobby Evans <ev...@yahoo-inc.com>.
You are correct. Rebalance updates the state so when mk-assignments is
called the scheduler does not see any workers associated with that
topology. It then will proceeded to reschedule it wherever the scheduler
thinks is best. Storm does not do any dynamic rebalancing yet. But if
you have a killer algorithm for it I would be happy to help update it so
that it could.
―Bobby
On 3/21/14, 1:18 AM, "hilfi alkaff" <hi...@gmail.com> wrote:
>Walk me through if I'm understanding the code correctly. I'm still very
>new
>to clojure. In cluster.clj, update-storm! just change the internal state
>variables, but it has not actually changed the state of the cluster? The
>actual changes happen at mk-assignments when it calls assignSlots?
>
>Additionally, storm does not perform any dynamic rebalancing in the fly
>correct? Or has there been a plan to do that? Thanks.
>
>
>On Wed, Mar 19, 2014 at 12:18 AM, 杨晓青 <xq...@gmail.com> wrote:
>
>> in file /storm-core/src/clj/backtype/storm/cluster.clj
>>
>>
>> 2014-03-19 11:42 GMT+08:00 hilfi alkaff <al...@illinois.edu>:
>>
>> > Hi,
>> >
>> > I would like to know how does Storm currently perform rebalance. I
>>see in
>> > *./storm-core/src/clj/backtype/storm/daemon/nimbus.clj* there's a
>> > *do-rebalance* method that will call *update-storm!* (Which I assume
>> > changes the state of Storm) but I don't see where the method
>> > *update-storm!*is being defined.
>> >
>> > Thanks,
>> >
>> > --
>> > Hilfi M. Alkaff
>> >
>>
>>
>>
>> --
>> -------------------------------------------
>> best regards
>>
>> 杨晓青
>>
>
>
>
>--
>~Hilfi Alkaff~
Re: Storm rebalance details.
Posted by hilfi alkaff <hi...@gmail.com>.
Walk me through if I'm understanding the code correctly. I'm still very new
to clojure. In cluster.clj, update-storm! just change the internal state
variables, but it has not actually changed the state of the cluster? The
actual changes happen at mk-assignments when it calls assignSlots?
Additionally, storm does not perform any dynamic rebalancing in the fly
correct? Or has there been a plan to do that? Thanks.
On Wed, Mar 19, 2014 at 12:18 AM, 杨晓青 <xq...@gmail.com> wrote:
> in file /storm-core/src/clj/backtype/storm/cluster.clj
>
>
> 2014-03-19 11:42 GMT+08:00 hilfi alkaff <al...@illinois.edu>:
>
> > Hi,
> >
> > I would like to know how does Storm currently perform rebalance. I see in
> > *./storm-core/src/clj/backtype/storm/daemon/nimbus.clj* there's a
> > *do-rebalance* method that will call *update-storm!* (Which I assume
> > changes the state of Storm) but I don't see where the method
> > *update-storm!*is being defined.
> >
> > Thanks,
> >
> > --
> > Hilfi M. Alkaff
> >
>
>
>
> --
> -------------------------------------------
> best regards
>
> 杨晓青
>
--
~Hilfi Alkaff~
Re: Storm rebalance details.
Posted by 杨晓青 <xq...@gmail.com>.
in file /storm-core/src/clj/backtype/storm/cluster.clj
2014-03-19 11:42 GMT+08:00 hilfi alkaff <al...@illinois.edu>:
> Hi,
>
> I would like to know how does Storm currently perform rebalance. I see in
> *./storm-core/src/clj/backtype/storm/daemon/nimbus.clj* there's a
> *do-rebalance* method that will call *update-storm!* (Which I assume
> changes the state of Storm) but I don't see where the method
> *update-storm!*is being defined.
>
> Thanks,
>
> --
> Hilfi M. Alkaff
>
--
-------------------------------------------
best regards
杨晓青