You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2017/03/09 11:46:36 UTC
Automating updates in process groups through templates
Good morning. We are developing NiFi workflows with a few key strategic
goals in mind, one of which is this: change common workflows in one place
only, and have those changes ripple through to all process
groups that employ that code.
More specifically: we have a complex but common workflow that we save as
template XYZ. That template is used within Process Group ABC, DEF, and XYZ.
We need to make improvements to the template, and want those improvements
to be picked up automatically by each of the process groups. How do we make
that happen?
Thank you in advance for your insights.
Jim
Re: Automating updates in process groups through templates
Posted by James McMahon <js...@gmail.com>.
Thank you Joe and Oleg. Reference-able Process Groups sounds like it would
address what we intend to do, without using templates. Which would be fine
and which would presumably serve the same purpose. How far off in the
future is this feature? Six months? A year?
NiFi has a rich and powerful tool set. In that respect it seems very
mature. From what little I understand about it to date it seems like the
time is right to take it to the next level in terms of "dev ops-y" like
capabilities. The ability to make changes in one place and automatically
pick up those changes as you described above (similar feature
to disseminate changes in templates would be nice too). Also, the ability
to promote select process groups in a master workflow from Dev to Test to
Int to Prod. I have much to learn about these things, and am happy to hear
that you are considering them too. Thank you again for your insights.
On Thu, Mar 9, 2017 at 7:14 AM, Joe Witt <jo...@gmail.com> wrote:
> Hello
>
> We have this concept or a very similar concept documented here as a
> feature proposal.
> https://cwiki.apache.org/confluence/display/NIFI/
> Reference-able+Process+Groups
> It hasn't been worked on yet but will be a really cool thing. The
> basic gist is that you can create a process group that acts as a
> callable function which you can then place in one or many places on
> the flow. Ensures you only have one thing/place to maintain for some
> critical logic step and it is more efficient on the system from a
> threading pov.
>
> We also have the concept of flow versioning
> https://cwiki.apache.org/confluence/display/NIFI/
> Configuration+Management+of+Flows
> which may also offer a similar behavior except it would be more like
> you described originally in that there would actually be multiple
> instances of that group but with it being versioned the other
> instances can be easily kept in sync.
>
> Thanks
> Joe
>
> On Thu, Mar 9, 2017 at 7:01 AM, James McMahon <js...@gmail.com>
> wrote:
> > I am happy to do this Oleg. Thank you very much for your reply.
> >
> > Cheers,
> >
> > Jim
> >
> > On Thu, Mar 9, 2017 at 6:56 AM, Oleg Zhurakousky
> > <oz...@hortonworks.com> wrote:
> >>
> >> Jim
> >>
> >> I see where you’re going with this and I do believe that it is a
> valuable
> >> enterprise feature. Unfortunately, template XML files are not the
> bootstrap
> >> source of the flow such as flow.gz file. They are only used for
> >> save/move/migrate/etc., which means any modifications to them are not
> going
> >> to be automatically picked up.
> >> But as I said, I do personally believe that something like this would be
> >> extremely valuable and could potentially be put on NiFi roadmap. Would
> you
> >> mind raising a Feature Request - https://issues.apache.org/
> jira/browse/NIFI
> >> so we can at least track it and gage the public interest?
> >>
> >> Cheers
> >> Oleg
> >>
> >> > On Mar 9, 2017, at 6:46 AM, James McMahon <js...@gmail.com>
> wrote:
> >> >
> >> > Good morning. We are developing NiFi workflows with a few key
> strategic
> >> > goals in mind, one of which is this: change common workflows in one
> place
> >> > only, and have those changes ripple through to all process groups that
> >> > employ that code.
> >> >
> >> > More specifically: we have a complex but common workflow that we save
> as
> >> > template XYZ. That template is used within Process Group ABC, DEF,
> and XYZ.
> >> > We need to make improvements to the template, and want those
> improvements to
> >> > be picked up automatically by each of the process groups. How do we
> make
> >> > that happen?
> >> >
> >> > Thank you in advance for your insights.
> >> >
> >> > Jim
> >>
> >
>
Re: Automating updates in process groups through templates
Posted by Joe Witt <jo...@gmail.com>.
Hello
We have this concept or a very similar concept documented here as a
feature proposal.
https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups
It hasn't been worked on yet but will be a really cool thing. The
basic gist is that you can create a process group that acts as a
callable function which you can then place in one or many places on
the flow. Ensures you only have one thing/place to maintain for some
critical logic step and it is more efficient on the system from a
threading pov.
We also have the concept of flow versioning
https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows
which may also offer a similar behavior except it would be more like
you described originally in that there would actually be multiple
instances of that group but with it being versioned the other
instances can be easily kept in sync.
Thanks
Joe
On Thu, Mar 9, 2017 at 7:01 AM, James McMahon <js...@gmail.com> wrote:
> I am happy to do this Oleg. Thank you very much for your reply.
>
> Cheers,
>
> Jim
>
> On Thu, Mar 9, 2017 at 6:56 AM, Oleg Zhurakousky
> <oz...@hortonworks.com> wrote:
>>
>> Jim
>>
>> I see where you’re going with this and I do believe that it is a valuable
>> enterprise feature. Unfortunately, template XML files are not the bootstrap
>> source of the flow such as flow.gz file. They are only used for
>> save/move/migrate/etc., which means any modifications to them are not going
>> to be automatically picked up.
>> But as I said, I do personally believe that something like this would be
>> extremely valuable and could potentially be put on NiFi roadmap. Would you
>> mind raising a Feature Request - https://issues.apache.org/jira/browse/NIFI
>> so we can at least track it and gage the public interest?
>>
>> Cheers
>> Oleg
>>
>> > On Mar 9, 2017, at 6:46 AM, James McMahon <js...@gmail.com> wrote:
>> >
>> > Good morning. We are developing NiFi workflows with a few key strategic
>> > goals in mind, one of which is this: change common workflows in one place
>> > only, and have those changes ripple through to all process groups that
>> > employ that code.
>> >
>> > More specifically: we have a complex but common workflow that we save as
>> > template XYZ. That template is used within Process Group ABC, DEF, and XYZ.
>> > We need to make improvements to the template, and want those improvements to
>> > be picked up automatically by each of the process groups. How do we make
>> > that happen?
>> >
>> > Thank you in advance for your insights.
>> >
>> > Jim
>>
>
Re: Automating updates in process groups through templates
Posted by James McMahon <js...@gmail.com>.
I am happy to do this Oleg. Thank you very much for your reply.
Cheers,
Jim
On Thu, Mar 9, 2017 at 6:56 AM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:
> Jim
>
> I see where you’re going with this and I do believe that it is a valuable
> enterprise feature. Unfortunately, template XML files are not the bootstrap
> source of the flow such as flow.gz file. They are only used for
> save/move/migrate/etc., which means any modifications to them are not going
> to be automatically picked up.
> But as I said, I do personally believe that something like this would be
> extremely valuable and could potentially be put on NiFi roadmap. Would you
> mind raising a Feature Request - https://issues.apache.org/
> jira/browse/NIFI so we can at least track it and gage the public interest?
>
> Cheers
> Oleg
>
> > On Mar 9, 2017, at 6:46 AM, James McMahon <js...@gmail.com> wrote:
> >
> > Good morning. We are developing NiFi workflows with a few key strategic
> goals in mind, one of which is this: change common workflows in one place
> only, and have those changes ripple through to all process groups that
> employ that code.
> >
> > More specifically: we have a complex but common workflow that we save as
> template XYZ. That template is used within Process Group ABC, DEF, and XYZ.
> We need to make improvements to the template, and want those improvements
> to be picked up automatically by each of the process groups. How do we make
> that happen?
> >
> > Thank you in advance for your insights.
> >
> > Jim
>
>
Re: Automating updates in process groups through templates
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Jim
I see where you’re going with this and I do believe that it is a valuable enterprise feature. Unfortunately, template XML files are not the bootstrap source of the flow such as flow.gz file. They are only used for save/move/migrate/etc., which means any modifications to them are not going to be automatically picked up.
But as I said, I do personally believe that something like this would be extremely valuable and could potentially be put on NiFi roadmap. Would you mind raising a Feature Request - https://issues.apache.org/jira/browse/NIFI so we can at least track it and gage the public interest?
Cheers
Oleg
> On Mar 9, 2017, at 6:46 AM, James McMahon <js...@gmail.com> wrote:
>
> Good morning. We are developing NiFi workflows with a few key strategic goals in mind, one of which is this: change common workflows in one place only, and have those changes ripple through to all process groups that employ that code.
>
> More specifically: we have a complex but common workflow that we save as template XYZ. That template is used within Process Group ABC, DEF, and XYZ. We need to make improvements to the template, and want those improvements to be picked up automatically by each of the process groups. How do we make that happen?
>
> Thank you in advance for your insights.
>
> Jim
RE: Automating updates in process groups through templates
Posted by "Carlos Manuel Fernandes (DSI)" <ca...@cgd.pt>.
James, one idea, if your common workflow was wrapped with HandleHttpRequest and HandleHttpResponse you can reuse them (calling with invokeHttp) since you pass all the parameters in the request and your workflow use them as attributes. Your common workflow behaves like a Micro Service.
Of course , the feature Reference-able+Process+Groups suggested by Joe Witt its more clean and readable solution.
Carlos
From: James McMahon [mailto:jsmcmahon3@gmail.com]
Sent: quinta-feira, 9 de Março de 2017 11:47
To: users@nifi.apache.org
Subject: Automating updates in process groups through templates
Good morning. We are developing NiFi workflows with a few key strategic goals in mind, one of which is this: change common workflows in one place only, and have those changes ripple through to all process groups that employ that code.
More specifically: we have a complex but common workflow that we save as template XYZ. That template is used within Process Group ABC, DEF, and XYZ. We need to make improvements to the template, and want those improvements to be picked up automatically by each of the process groups. How do we make that happen?
Thank you in advance for your insights.
Jim