Curious about Best Practices for Deployment of Workflows

[Ken Swanson <ks...@1904labs.com>]

Hi all,

My group is getting started with NiFi and I'm hoping to draw on people's
experience. I was curious if there were any best practices for NiFi when
deploying workflows. When I deploy new code to production, there's often
pull requests, code review, and then CI/CD to actually put it in
production. Is there anything similar for workflows, or do people have
solutions that they use?

I'd love it if there was a way for someone to make a local change to a
workflow/process group, push that up to the registry, and then a reviewer
see a diff of the change, approve it, and then it get automatically put
onto master. Is this something people tend to do with NiFi?

-Ken

Re: Curious about Best Practices for Deployment of Workflows

[Andy LoPresto <al...@apache.org>]

Ken, 

There are a lot of disjoint resources that discuss some of these concepts, but they are not well-organized at the moment. I would recommend Kevin Doran’s presentation at Dataworks Summit 2018 [1] as a good starting point, specifically including some activity hooks which can automate much of the process you describe. Drew Lim has also recorded some videos showing features and workflows (I separate workflow [“the process of performing some manual activities against the NiFi ecosystem applications”] from dataflow [“the graph of components which runs within NiFi and ingests, routes, transforms, and persists data”]) [2] (scroll down to “Videos”). 

If others can chime in with their resources as well, we can collect these into a more useful format and hopefully enable more people to use these tools productively. 

[1] https://dataworkssummit.com/san-jose-2018/session/sdlc-with-apache-nifi/ <https://dataworkssummit.com/san-jose-2018/session/sdlc-with-apache-nifi/>
[2] https://nifi.apache.org/registry.html <https://nifi.apache.org/registry.html>


Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 18, 2019, at 12:53 PM, Ken Swanson <ks...@1904labs.com> wrote:
> 
> Hi all,
> 
> My group is getting started with NiFi and I'm hoping to draw on people's experience. I was curious if there were any best practices for NiFi when deploying workflows. When I deploy new code to production, there's often pull requests, code review, and then CI/CD to actually put it in production. Is there anything similar for workflows, or do people have solutions that they use?
> 
> I'd love it if there was a way for someone to make a local change to a workflow/process group, push that up to the registry, and then a reviewer see a diff of the change, approve it, and then it get automatically put onto master. Is this something people tend to do with NiFi?
> 
> -Ken