Adding a node to a cluster

[James McMahon <js...@gmail.com>]

A nifi cluster has been created using a CloudFormation script. The cluster
is running, and the customer now wishes to scale up the cluster by adding a
node. What is the optimal way to add a node to a running nifi cluster
established through CloudFormation? My instinct tells me to do it within
the confines of CloudFormation so that the new node isn't lost if
CloudFormation is re-run. But does that mean the active cluster needs to be
shutdown just to modify the CloudFormation scripts and restart it with the
new cluster configuration?

Has anyone successfully done this, and can offer some tips or a link to an
outline of the process?

Re: Adding a node to a cluster

[James McMahon <js...@gmail.com>]

This is an external zookeeper, Mein Herr.  And this team’s current thought
is that the easiest way to add a node is to modify their CloudFormation
script, and use it to tear down and re-standup their N-cluster + 1. I’m
wondering that there might be a less-disruptive way to add the “+1”. Is
there?

auf Wiedersehen
Jim

On Mon, Apr 11, 2022 at 3:51 PM Jorge Machado <jo...@me.com> wrote:

> Hey,
>
> Are you using the zookeeper internally or an external one. I think you
> need to check at least two things:
>
> * Zookeeper connections. The node needs to register into zookeeper.
> * The node needs to connect to the cluster and replicate the flow files
> let’s say so.
>
>
> Overall scaling up is ok, but scaling down it’s complicated you need to
> make sure that you disable the node and remove it from the cluster. You
> cannot simple tear it down as it has state.
>
> Mit freundlichen Grüßen
> Jorge
>
> > On 11. Apr 2022, at 15:19, James McMahon <js...@gmail.com> wrote:
> >
> > A nifi cluster has been created using a CloudFormation script. The
> cluster is running, and the customer now wishes to scale up the cluster by
> adding a node. What is the optimal way to add a node to a running nifi
> cluster established through CloudFormation? My instinct tells me to do it
> within the confines of CloudFormation so that the new node isn't lost if
> CloudFormation is re-run. But does that mean the active cluster needs to be
> shutdown just to modify the CloudFormation scripts and restart it with the
> new cluster configuration?
> >
> > Has anyone successfully done this, and can offer some tips or a link to
> an outline of the process?
>
>

Re: Adding a node to a cluster

[Jorge Machado <jo...@me.com>]

Hey, 

Are you using the zookeeper internally or an external one. I think you need to check at least two things: 

* Zookeeper connections. The node needs to register into zookeeper. 
* The node needs to connect to the cluster and replicate the flow files let’s say so. 


Overall scaling up is ok, but scaling down it’s complicated you need to make sure that you disable the node and remove it from the cluster. You cannot simple tear it down as it has state. 

Mit freundlichen Grüßen 
Jorge  

> On 11. Apr 2022, at 15:19, James McMahon <js...@gmail.com> wrote:
> 
> A nifi cluster has been created using a CloudFormation script. The cluster is running, and the customer now wishes to scale up the cluster by adding a node. What is the optimal way to add a node to a running nifi cluster established through CloudFormation? My instinct tells me to do it within the confines of CloudFormation so that the new node isn't lost if CloudFormation is re-run. But does that mean the active cluster needs to be shutdown just to modify the CloudFormation scripts and restart it with the new cluster configuration?
> 
> Has anyone successfully done this, and can offer some tips or a link to an outline of the process?