Rebuilding NiFi Cluster

[David Pliskin <pl...@outlook.com>]

Hi all,

I am currently running a NiFi setup configured as a single NiFi instance connecting both input/output ports to a remote progress group. The goal is to have the single instance handle disk I/O, and hand off CPU intensive work to members of a cluster. So far, this works wonderfully.

However, I am looking at periodically rebuilding all the machines the cluster is hosted on, causing 2 pain points;

1. Loading a preconfigured flow.tar file. Copying a flow.xml.gz file to a standalone instance sees the instance loading the diagram upon initialisation, but using a flow.tar in a cluster does not seem as easy. I have taken a configured flow.tar/flow.xml file and copied it into a blank cluster (1 manager, 0 nodes) flow.tar file, then attempted to have a fresh manager instance load the new flow.tar, but have not had any luck. NiFi will attempt to start, but then error out.

At the moment, I am circumventing this with a manual step by using the NiFi GUI to load a saved flow template. But, I would much prefer if I was able to use a preconfigured flow in my image build process

2. Remote progress group ports no longer exist, a problem seen by the standalone instance once the replacement cluster has been built. The remote process group disables all previously enabled ports, and claims they are no longer accessible. Also, the new ports found in the cluster are listed as "off". It looks like the ports are referenced using an ID, and that ID is different once the replacement cluster has been deployed.

This means I have to manually intervene by; deleting the old ports from the RPG, recreating links between the RPG and any follow-on processors, and then enabling the remote ports in the RPG block.

Ideally, I would like to be able to remove both manual steps from my deployment process, but am unsure if there are any supported mechanisms for what I'd like to achieve.

Cheers,
David