Dynamic reconfiguration for the nodes in Nimbus/Pacemaker clusters

[S G <sg...@gmail.com>]

Hi,

I was reading a zookeeper article:
https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html

And was wondering if a similar functionality exists for Nimbus/Pacemaker
Cluster.
It would be really nice if there is no need for restarting servers in the
Nimbus/Pacemaker cluster whenever a node exits or joins the above clusters.

Is this something already existing in storm?
Or we can raise a JIRA if it makes sense to have it in storm.

Thanks
SG

Re: Dynamic reconfiguration for the nodes in Nimbus/Pacemaker clusters

[S G <sg...@gmail.com>]

Thanks for the inputs Bobby.
https://issues.apache.org/jira/browse/STORM-2269 is there to track this.
I will also try to look into some code if I get time this week.

On Tue, Jan 3, 2017 at 6:42 AM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:

> There is nothing for that right now on pacemaker.  You can do it with
> nimbus so long as at least one of the original nodes is still up.  But in
> either case it would not be too difficult to make it all fully functional.
> The two critical pieces would be in giving the workers and daemons a way to
> reload these specific configs dynamically.  Then it would be documenting
> the order of operations to be sure nothing goes wrong.
> Adding Pacemaker Node(s).1) bring up the new node(s).2) update nimbus
> configs to start reading from the new nodes.3) update all of the worker
> nodes to let workers start writing to the new node.
> Removing Pacemaker Node(s)1) Shut down pacemaker nodes/update configs on
> workers (order should not matter so long as there are enough pacemaker
> nodes up to handle the load)2) update the nimbus configs to not try and
> read from the old nodes
> Adding new Nimbus Node(s).1) Bring up the new nimbus with the new
> config.2) update all of the other nodes (including any machines that
> clients come from) with new config (order does not matter)
> Removing Nimbus Node(s).Shut down the old nodes and update the configs on
> all the boxes in any order you want. This should just work so long as you
> have at least one nimbus node still up.
> If you want to file a JIRA for it that would be wonderful.  If you want to
> do some of the work on reloading the configs periodically that would be
> great too. - Bobby
>
>     On Monday, January 2, 2017 9:21 PM, S G <sg...@gmail.com>
> wrote:
>
>
>  Hi,
>
> I was reading a zookeeper article:
> https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html
>
> And was wondering if a similar functionality exists for Nimbus/Pacemaker
> Cluster.
> It would be really nice if there is no need for restarting servers in the
> Nimbus/Pacemaker cluster whenever a node exits or joins the above clusters.
>
> Is this something already existing in storm?
> Or we can raise a JIRA if it makes sense to have it in storm.
>
> Thanks
> SG
>
>
>
>

Re: Dynamic reconfiguration for the nodes in Nimbus/Pacemaker clusters

[Bobby Evans <ev...@yahoo-inc.com.INVALID>]

There is nothing for that right now on pacemaker.  You can do it with nimbus so long as at least one of the original nodes is still up.  But in either case it would not be too difficult to make it all fully functional.  The two critical pieces would be in giving the workers and daemons a way to reload these specific configs dynamically.  Then it would be documenting the order of operations to be sure nothing goes wrong.
Adding Pacemaker Node(s).1) bring up the new node(s).2) update nimbus configs to start reading from the new nodes.3) update all of the worker nodes to let workers start writing to the new node.
Removing Pacemaker Node(s)1) Shut down pacemaker nodes/update configs on workers (order should not matter so long as there are enough pacemaker nodes up to handle the load)2) update the nimbus configs to not try and read from the old nodes
Adding new Nimbus Node(s).1) Bring up the new nimbus with the new config.2) update all of the other nodes (including any machines that clients come from) with new config (order does not matter)
Removing Nimbus Node(s).Shut down the old nodes and update the configs on all the boxes in any order you want. This should just work so long as you have at least one nimbus node still up.
If you want to file a JIRA for it that would be wonderful.  If you want to do some of the work on reloading the configs periodically that would be great too. - Bobby 

    On Monday, January 2, 2017 9:21 PM, S G <sg...@gmail.com> wrote:
 

 Hi,

I was reading a zookeeper article:
https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html

And was wondering if a similar functionality exists for Nimbus/Pacemaker
Cluster.
It would be really nice if there is no need for restarting servers in the
Nimbus/Pacemaker cluster whenever a node exits or joins the above clusters.

Is this something already existing in storm?
Or we can raise a JIRA if it makes sense to have it in storm.

Thanks
SG