Configuration Management of Flows - Proposed Book of Work

[Gunjan Dave <gu...@gmail.com>]

Hello NiFi Team,
I understand that config mgmt of flows is part of the proposed road map.
Is there any tangible action which has started on this front? Any tentative
release plan even if very preliminary?
Any plans to integrate with Git internally within the framework itself to
version manage?
This will soon become a differentiating factor for choosing NiFi over other
products.

Re: Configuration Management of Flows - Proposed Book of Work

[Gunjan Dave <gu...@gmail.com>]

Thanks Joe, i'll try to digest the information.

In mean time, could you kindly help if there is any specific jira's or
links in terms of what has improved in NiFi 1.0 for using templates with
version control and diff tool, that alone at the moment should solve lot of
problems, i guess.

In the meantime will also wait for many more great features to come in
w.r.t to configuration management of flows.

On Mon, Aug 29, 2016, 1:05 PM Joe Witt <jo...@gmail.com> wrote:

> Gunjan
>
> We've long since supported the concept of flow templates.  These are
> powerful because they allow you to save, share, import already
> designed flows.  They have to date had their full potential limited by
> a few things:
>
> 1) The exported template was non-deterministic in terms of the XML it
> produced.  This made using typical version control and diff tools very
> difficult in terms of being able to quickly assess what has changed.
> This was addressed in the upcoming Apache NiFi 1.0 which is under
> vote.
>
> 2) The templates were too coupled to specific systems such as if you
> had a URL for a database then that URL was in the template which made
> the template less portable to say a dev environment which has a
> different URL.  This too has been improved in the upcoming version as
> it now supports a variable registry and for properties which take
> advantage of expression language statements they can now take
> advantage of variable registry entries.  We intend to do more there
> too [1].
>
> 3) We also need to provide a registry to make saving and sharing these
> templates easier than it is today [2].  This will allow centralized
> registries of templates that organizations can share between NiFi
> clusters and do things like support typical SDLC models.  There are
> many more things we can do with this connecting templates to the
> extensions they reference and supporting things like multiple versions
> of those extensions [3]
>
> Finally, regarding configuration management in general there is no
> reason why at this point we cannot simply always create a commit/diff
> of the flow as changes are made and have them stored in a version
> control system.  We can then do things like tag data against the
> version of the flow it ran through and rollback to a given
> configuration state [4].
>
> Hopefully this helps give you a sense of the ideas, progress, and
> discussions that have occurred to date.
>
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
> [2]
> https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
> [3]
> https://cwiki.apache.org/confluence/display/NIFI/Multiple+Versions+of+the+Same+Extension
> [4]
> https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows
>
>
> Thanks
> Joe
>
> On Sun, Aug 28, 2016 at 11:48 PM, Gunjan Dave
> <gu...@gmail.com> wrote:
> > Hello NiFi Team,
> > I understand that config mgmt of flows is part of the proposed road map.
> > Is there any tangible action which has started on this front? Any
> tentative
> > release plan even if very preliminary?
> > Any plans to integrate with Git internally within the framework itself to
> > version manage?
> > This will soon become a differentiating factor for choosing NiFi over
> other
> > products.
> >
> >
>

Re: Configuration Management of Flows - Proposed Book of Work

[Joe Witt <jo...@gmail.com>]

Gunjan

We've long since supported the concept of flow templates.  These are
powerful because they allow you to save, share, import already
designed flows.  They have to date had their full potential limited by
a few things:

1) The exported template was non-deterministic in terms of the XML it
produced.  This made using typical version control and diff tools very
difficult in terms of being able to quickly assess what has changed.
This was addressed in the upcoming Apache NiFi 1.0 which is under
vote.

2) The templates were too coupled to specific systems such as if you
had a URL for a database then that URL was in the template which made
the template less portable to say a dev environment which has a
different URL.  This too has been improved in the upcoming version as
it now supports a variable registry and for properties which take
advantage of expression language statements they can now take
advantage of variable registry entries.  We intend to do more there
too [1].

3) We also need to provide a registry to make saving and sharing these
templates easier than it is today [2].  This will allow centralized
registries of templates that organizations can share between NiFi
clusters and do things like support typical SDLC models.  There are
many more things we can do with this connecting templates to the
extensions they reference and supporting things like multiple versions
of those extensions [3]

Finally, regarding configuration management in general there is no
reason why at this point we cannot simply always create a commit/diff
of the flow as changes are made and have them stored in a version
control system.  We can then do things like tag data against the
version of the flow it ran through and rollback to a given
configuration state [4].

Hopefully this helps give you a sense of the ideas, progress, and
discussions that have occurred to date.


[1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
[2] https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
[3] https://cwiki.apache.org/confluence/display/NIFI/Multiple+Versions+of+the+Same+Extension
[4] https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows


Thanks
Joe

On Sun, Aug 28, 2016 at 11:48 PM, Gunjan Dave
<gu...@gmail.com> wrote:
> Hello NiFi Team,
> I understand that config mgmt of flows is part of the proposed road map.
> Is there any tangible action which has started on this front? Any tentative
> release plan even if very preliminary?
> Any plans to integrate with Git internally within the framework itself to
> version manage?
> This will soon become a differentiating factor for choosing NiFi over other
> products.
>
>