You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Manish Gupta 8 <mg...@sapient.com> on 2016/09/27 18:27:11 UTC
NiFi Cluster mode
Hello Everyone,
In one of our project, we only have one large box available for running NiFi in production (to start with). This is due to some data center related issues (availability of space etc.). Later, if required, we will move to a multi-node cluster.
We are thinking about - Should we set up NiFi to run in Clustered mode from the beginning i.e. NCM + Slave node on single machine or should we set it up as single-node deployment. Some of the considerations are:
* How difficult is it to move from single node deployment (without running NCM) to a multi-node cluster? I mean, can we simply make some configuration settings and run NCM to existing installation and add multiple nodes to it? Should we worry about any potential data/configuration loss while doing such an upgrade?
* Is NCM resource intensive?
* Any other factor we should consider?
Thanks,
Manish
RE: NiFi Cluster mode
Posted by Manish Gupta 8 <mg...@sapient.com>.
Thanks Andy. Sorry, I forgot to mention – we are on 0.7 now, and will be moving to 1.0 after some time. Also, we have 128 GB machine.
So, we will go with multi-node setup only.
Regards,
Manish
From: Andy LoPresto [mailto:alopresto@apache.org]
Sent: Wednesday, September 28, 2016 12:06 AM
To: users@nifi.apache.org
Subject: Re: NiFi Cluster mode
Hi Manish,
With NiFi 1.0.0, the NCM model has been replaced by “zero master clustering” or “ZMC” [1]. This means an arbitrary number of nodes can run in the cluster, and there is no longer a SPOF. With ZMC, all connected nodes provide the UI and replicate changes to the flow to all other nodes. If a node is lost, a new coordinating node is elected via ZooKeeper.
In your case, if the machine available really is “large”, I would definitely recommend setting up a cluster on the single node and migrating or expanding it when possible. Migrating from a single node to a multi-node cluster is just a matter of configuration changes, and you will not lose any data. However, if you configure the application as a cluster from the beginning, you can seamlessly introduce new nodes to the cluster as they become provisioned with minimal changes to the running instances. Just ensure that you have the RAM available to run decent Java heaps for both instances, and follow the configuration best practices in the NiFi Admin Guide [2] for open file handles, etc.
[1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Sep 27, 2016, at 11:27 AM, Manish Gupta 8 <mg...@sapient.com>> wrote:
Hello Everyone,
In one of our project, we only have one large box available for running NiFi in production (to start with). This is due to some data center related issues (availability of space etc.). Later, if required, we will move to a multi-node cluster.
We are thinking about - Should we set up NiFi to run in Clustered mode from the beginning i.e. NCM + Slave node on single machine or should we set it up as single-node deployment. Some of the considerations are:
• How difficult is it to move from single node deployment (without running NCM) to a multi-node cluster? I mean, can we simply make some configuration settings and run NCM to existing installation and add multiple nodes to it? Should we worry about any potential data/configuration loss while doing such an upgrade?
• Is NCM resource intensive?
• Any other factor we should consider?
Thanks,
Manish
Re: NiFi Cluster mode
Posted by Andy LoPresto <al...@apache.org>.
Hi Manish,
With NiFi 1.0.0, the NCM model has been replaced by “zero master clustering” or “ZMC” [1]. This means an arbitrary number of nodes can run in the cluster, and there is no longer a SPOF. With ZMC, all connected nodes provide the UI and replicate changes to the flow to all other nodes. If a node is lost, a new coordinating node is elected via ZooKeeper.
In your case, if the machine available really is “large”, I would definitely recommend setting up a cluster on the single node and migrating or expanding it when possible. Migrating from a single node to a multi-node cluster is just a matter of configuration changes, and you will not lose any data. However, if you configure the application as a cluster from the beginning, you can seamlessly introduce new nodes to the cluster as they become provisioned with minimal changes to the running instances. Just ensure that you have the RAM available to run decent Java heaps for both instances, and follow the configuration best practices in the NiFi Admin Guide [2] for open file handles, etc.
[1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Sep 27, 2016, at 11:27 AM, Manish Gupta 8 <mg...@sapient.com> wrote:
>
> Hello Everyone,
>
> In one of our project, we only have one large box available for running NiFi in production (to start with). This is due to some data center related issues (availability of space etc.). Later, if required, we will move to a multi-node cluster.
>
> We are thinking about - Should we set up NiFi to run in Clustered mode from the beginning i.e. NCM + Slave node on single machine or should we set it up as single-node deployment. Some of the considerations are:
> · How difficult is it to move from single node deployment (without running NCM) to a multi-node cluster? I mean, can we simply make some configuration settings and run NCM to existing installation and add multiple nodes to it? Should we worry about any potential data/configuration loss while doing such an upgrade?
> · Is NCM resource intensive?
> · Any other factor we should consider?
>
> Thanks,
> Manish