Cluster setup in multiple data centers

[Vijay Chhipa <vc...@apple.com>]

Hello 

We are putting NiFi in production. To ensure that there is no single point of failure we have two options. 

a) Setup a single NiFi cluster with nodes distributed across data centers. 
b) Setup a cluster in one data center and for failover have a hot stand-by cluster in another data center 

What are the pros and cons of each approach?

Are there other options?

Any feedback based on your experience will be highly useful. 

Thank you, 

Vijay

Re: Cluster setup in multiple data centers

[Mark Payne <ma...@hotmail.com>]

Hi Vijay,

You may run into some problems if you try to build a single NiFi cluster that spans multiple
data centers. NiFi depends on ZooKeeper to perform leader election to determine which node
is the Cluster Coordinator and which is the Primary Node. ZooKeeper guidance indicates that you
should not use it for cross-data center communication. Additionally, if ZooKeeper does still work
well for you cross-data center you'll likely see UI behaving sluggishly, as all of the requests would
have to be replicated across the data centers instead of just across a LAN. That said, I've never actually
attempted to do this, so you're welcome to try :)

However, I would lean toward having a second cluster. The NiFi Flow Registry [1][2] should help to make
this fairly painless.

Thanks
-Mark

[1] https://nifi.apache.org/registry.html
[2] https://bryanbende.com/development/2018/01/19/apache-nifi-how-do-i-deploy-my-flow


On Oct 22, 2018, at 12:36 AM, Vijay Chhipa <vc...@apple.com>> wrote:

Hello

We are putting NiFi in production. To ensure that there is no single point of failure we have two options.

a) Setup a single NiFi cluster with nodes distributed across data centers.
b) Setup a cluster in one data center and for failover have a hot stand-by cluster in another data center

What are the pros and cons of each approach?

Are there other options?

Any feedback based on your experience will be highly useful.

Thank you,

Vijay