Best practices for pushing to production

[Stéphane Maarek <st...@gmail.com>]

Hi,

How should we proceed to promote flow from dev to test to prod?
Basically, our stakeholders are not happy with having people modifying
anything in production (so it's a read-only environment), and basically
want people to draft / edit their flow in dev, push them to test, and then
to prod

I think this has a couple of challenges, such as :

0) How do you capture all the changes that were made? Is it a template, or
a list of API calls, or something else?
1) How do you programatically promote stuff from dev to test to prod? Stuff
that change between these environments are probably hostnames, etc
2) How do you edit a flow that's already been promoted to prod?
3) How to ensure that no data is left "hanging" in between processors
before a change to the flow? Or is it fine?

Interested in knowing how other companies do it, or manage their NiFi
cluster in production (what can be changed vs what can't)

Cheers,
Stephane

Re: Best practices for pushing to production

[Andy LoPresto <al...@apache.org>]

Hi Stéphane,

This is a request that has grown popular recently. NiFi was not initially designed with environment promotion in mind, so it is something we are currently investigating and trying to address.

The development/QA/production environment promotion process [1] (sometimes referred to as "SDLC" or "D2P" in conversation) is a topic of much discussion amongst the NiFi development community. Currently, there are plans to improve this process in a future release. For now, I will discuss some common behaviors/workflows that we have seen.

* The $NIFI_HOME/conf/flow.xml.gz file contains the entire flow serialized to XML. This file contains all processor configuration values, even sensitive values (encrypted). With the new Variable Registry [2] effort, you can refer to environment-specific variables transparently, and promote the same flow between environments without having to update specific values in the flow itself.
* The XML flow definition or specific templates can be committed and versioned using Git (or any other source code control tool). Recent improvements like "deterministic template diffs” [3] have made this versioning easier.
* The NiFi REST API [4] can be used to "automate" the deployment of a template or flow to an instance of NiFi.
* A script (Groovy, Python, etc.) could be used to integrate with both your source code control tool and your various NiFi instances to semi-automate this process (i.e. tap into Git hooks detecting a commit, and promote automatically to the next environment), but you probably want some human interaction to remain for verification at each state.
* To ensure that no “data is left hanging”, you can stop your input sources and allow the system to “drain” (i.e. check that all queues are empty before stopping the application, if replacing the flow.xml.gz file). If adding a new template, you can add this during normal operation and then configure and start the new flow while the old continues to run.

We understand that the current state of NiFi is not ideal for the promotion of the flow between dev/QA/prod environments. There are ongoing efforts to improve this, but I can't describe anything concrete at this time. If these points raise specific questions or you think of something else, please follow up.

[1] https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows
[2] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry <https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry>
[3] https://issues.apache.org/jira/browse/NIFI-826 <https://issues.apache.org/jira/browse/NIFI-826>
[4] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html


Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 11, 2016, at 7:37 PM, Stéphane Maarek <st...@gmail.com> wrote:
> 
> Hi,
> 
> How should we proceed to promote flow from dev to test to prod?
> Basically, our stakeholders are not happy with having people modifying anything in production (so it's a read-only environment), and basically want people to draft / edit their flow in dev, push them to test, and then to prod
> 
> I think this has a couple of challenges, such as :
> 
> 0) How do you capture all the changes that were made? Is it a template, or a list of API calls, or something else?
> 1) How do you programatically promote stuff from dev to test to prod? Stuff that change between these environments are probably hostnames, etc
> 2) How do you edit a flow that's already been promoted to prod?
> 3) How to ensure that no data is left "hanging" in between processors before a change to the flow? Or is it fine?
> 
> Interested in knowing how other companies do it, or manage their NiFi cluster in production (what can be changed vs what can't)
> 
> Cheers,
> Stephane