You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by anshu shukla <an...@gmail.com> on 2017/04/15 11:33:45 UTC
Spout resume after rebalance
Hello ,
I have a doubt that How storm identifies that on rebalancing when spout
should start emitting msgs.
Say in case I gave a wait time of 0 (-w 0) then rebalance will start
without pausing spout and msgs between will be lost.
*(Inflight msgs are lost but what about new incoming msgs ?)*
In short:
*How does spout confirms that all downstream bolts are ready to receive
msgs after rebalance and spout should start emitting msgs?*
--
Thanks & Regards,
Anshu Shukla
Re: Spout resume after rebalance
Posted by anshu shukla <an...@gmail.com>.
Thanks a lot, this is really helpful.
I will read the code logic and may again need to revert back if required.
- Mainly I have to look that *how spout* came to know that *mkAssignments
in [3] is complete. [Doubt- either it waits for some internal __system ack
or just start emitting]*
On Mon, Apr 17, 2017 at 9:41 PM, Arun Mahadevan <ar...@apache.org> wrote:
> During a rebalance [1] nimbus resets the component_executor map and
> arranges for a “DO_REBALANCE” to be triggered after a delay (see [2]).
>
>
>
> When nimbus receives the “DO_REBALANCE”, it does the actual rebalance
> where it computes and makes the new assignments [3].
>
>
>
> I would assume the spouts/bolts stops sometime after [1] and resumes only
> after mkAssignments in [3] is complete.
>
>
>
> Thanks,
>
> Arun
>
>
>
> [1] https://github.com/apache/storm/blob/master/storm-
> server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2652
>
> [2] https://github.com/apache/storm/blob/master/storm-
> server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L260
>
> [3] https://github.com/apache/storm/blob/master/storm-
> server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L1181
>
>
>
>
>
> *From: *anshu shukla <an...@gmail.com>
> *Reply-To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Date: *Monday, April 17, 2017 at 8:11 PM
> *To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Subject: *Re: Spout resume after rebalance
>
>
>
> *Does spout receives any special msg and only then it resumes to emit
> msgs ? *
>
>
>
> On Sat, Apr 15, 2017 at 5:03 PM, anshu shukla <an...@gmail.com>
> wrote:
>
> Hello ,
>
>
>
> I have a doubt that How storm identifies that on rebalancing when spout
> should start emitting msgs.
>
> Say in case I gave a wait time of 0 (-w 0) then rebalance will start
> without pausing spout and msgs between will be lost.
>
>
>
> *(Inflight msgs are lost but what about new incoming msgs ?)*
>
> In short:
>
> *How does spout confirms that all downstream bolts are ready to receive
> msgs after rebalance and spout should start emitting msgs?*
>
>
>
> --
>
> Thanks & Regards,
> Anshu Shukla
>
>
>
>
>
> --
>
> Thanks & Regards,
> Anshu Shukla
>
--
Thanks & Regards,
Anshu Shukla
Re: Spout resume after rebalance
Posted by Arun Mahadevan <ar...@apache.org>.
During a rebalance [1] nimbus resets the component_executor map and arranges for a “DO_REBALANCE” to be triggered after a delay (see [2]).
When nimbus receives the “DO_REBALANCE”, it does the actual rebalance where it computes and makes the new assignments [3].
I would assume the spouts/bolts stops sometime after [1] and resumes only after mkAssignments in [3] is complete.
Thanks,
Arun
[1] https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2652
[2] https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L260
[3] https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L1181
From: anshu shukla <an...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Monday, April 17, 2017 at 8:11 PM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Spout resume after rebalance
Does spout receives any special msg and only then it resumes to emit msgs ?
On Sat, Apr 15, 2017 at 5:03 PM, anshu shukla <an...@gmail.com> wrote:
Hello ,
I have a doubt that How storm identifies that on rebalancing when spout should start emitting msgs.
Say in case I gave a wait time of 0 (-w 0) then rebalance will start without pausing spout and msgs between will be lost.
(Inflight msgs are lost but what about new incoming msgs ?)
In short:
How does spout confirms that all downstream bolts are ready to receive msgs after rebalance and spout should start emitting msgs?
--
Thanks & Regards,
Anshu Shukla
--
Thanks & Regards,
Anshu Shukla
Re: Spout resume after rebalance
Posted by anshu shukla <an...@gmail.com>.
*Does spout receives any special msg and only then it resumes to emit msgs
? *
On Sat, Apr 15, 2017 at 5:03 PM, anshu shukla <an...@gmail.com>
wrote:
> Hello ,
>
> I have a doubt that How storm identifies that on rebalancing when spout
> should start emitting msgs.
> Say in case I gave a wait time of 0 (-w 0) then rebalance will start
> without pausing spout and msgs between will be lost.
>
> *(Inflight msgs are lost but what about new incoming msgs ?)*
> In short:
> *How does spout confirms that all downstream bolts are ready to receive
> msgs after rebalance and spout should start emitting msgs?*
>
> --
> Thanks & Regards,
> Anshu Shukla
>
--
Thanks & Regards,
Anshu Shukla