Pull from NiFi Registry on startup in a new instance of NiFi

["Kessler, Jon" <Ke...@varentech.com>]

I'm interested in being able to configure NiFi in a way that it will, upon startup, create a process group using the latest (or otherwise specified) flow that is stored within a NiFi Registry (probably something in nifi.properties). If it fails to connect to/pull from the registry then I want that NiFi to fail to start. I think this would be useful when it comes to deploying NiFi in a container so that you can have it start with a specific flow without having to have it join an active NiFi cluster. That process group (or groups if we want to get fancy and allow more than one) should be configured so that version control with the registry is already set up as well.

This behavior should only occur if the NiFi being started currently has an empty flow. This should prevent changes to the graph from being stomped on if that NiFi is restarted later on.

Just wanted to solicit thoughts on such a thing.

Re: Pull from NiFi Registry on startup in a new instance of NiFi

[Andy LoPresto <al...@apache.org>]

Jon, 

I think your idea is interesting but there are a lot of moving parts here with negotiating a connection, validating security, and defining how the communication between NiFi and Registry would work. Those definitions are not stored in the nifi,properties file, and introducing them there would be difficult. 

There is, however, a simpler solution to your request. You can populate a flow.xml.gz in your NiFi container so that it has that flow definition without needing to join an existing cluster. You can make that flow definition static or dynamic by using the NiFi CLI Toolkit to retrieve a specific flow definition from the registry instance as part of your container setup process. 

This should meet the requirements you’ve listed below. 


Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 20, 2020, at 12:00 PM, Kessler, Jon <Ke...@varentech.com> wrote:
> 
> I'm interested in being able to configure NiFi in a way that it will, upon startup, create a process group using the latest (or otherwise specified) flow that is stored within a NiFi Registry (probably something in nifi.properties). If it fails to connect to/pull from the registry then I want that NiFi to fail to start. I think this would be useful when it comes to deploying NiFi in a container so that you can have it start with a specific flow without having to have it join an active NiFi cluster. That process group (or groups if we want to get fancy and allow more than one) should be configured so that version control with the registry is already set up as well.
> 
> This behavior should only occur if the NiFi being started currently has an empty flow. This should prevent changes to the graph from being stomped on if that NiFi is restarted later on.
> 
> Just wanted to solicit thoughts on such a thing.