You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Anil Rai <an...@gmail.com> on 2018/03/13 14:48:48 UTC
Reuse of templates and formatting functions
Team,
My question is around re-use.
Templates : We are creating flows for different use cases. Most of them
have a requirement for retry. Say the API call fails, we would like to
retry 3 time after certain interval before giving up. We have created a
retry template that works perfect.
Q: If I have to use this template, then i have to import this template in
all the use case flows. If we have a change to this template, then i would
be forced to go to every flow that uses this template and make that change.
Date, String, Number formatting : I have seen couple of ways of doing this
formatting. Like in UpdateAttribute, Jolt transformation, ExecuteScript.
Q: Like in Java, can we have this utility in one place that can be used by
all my flows at run time by passing the required Parameters?
Thanks
Anil
Re: Reuse of templates and formatting functions
Posted by Ed B <bd...@gmail.com>.
Nothing is impossible in our Universe!
Although I would rather agree with the approach Bryan mentioned below, it
is still possible to implement a wormhole.
I could successfully create an example (see attached).
Downsides of this implementation:
- you have to keep multiple remote output ports (one per for each calling
flow);
- flowfiles will be using site-to-site twice: to enter to the reusable
module and to leave it.
- on output - flowfiles will go to ALL flows having relationship with RGPs
using the same remote input port, so additional filter is required to avoid
flowfiles coming from "neighbors"
Use responsibly :)
On Tue, Mar 13, 2018 at 1:31 PM Anil Rai <an...@gmail.com> wrote:
> Thanks Bryan. That helps.
> We were also thinking of exposing standard formatting functions as an API
> that other data flows can call and use.
>
> Regards
> Anil
>
>
> On Tue, Mar 13, 2018 at 1:13 PM, Bryan Bende <bb...@gmail.com> wrote:
>
>> Anil,
>>
>> It sounds like what you are interested in would be some combination of
>> the referenceable process groups [1] or worm-hole connections [2],
>> which were ideas that were discussed, but haven't been implemented.
>>
>> You might be able to make your situation slightly better by using the
>> NiFi Registry instead of templates...
>>
>> As an example, you could design a stand-alone process group for your
>> retry logic and place it under version control in the registry. Then
>> you import this flow from the registry into various points of your
>> overall flow.
>>
>> You could then go back to the original standalone process group and
>> make a change to the logic and save version 2 to the registry. At this
>> point all the other places you imported it to will show an upgrade and
>> you can change their version to v2 to bring in the change.
>>
>> It still requires manual upgrading of all the versioned process
>> groups, but at least they are all tied back to the same version, where
>> as templates are not linked to anything once they instantiated.
>>
>> -Bryan
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups
>> [2] https://cwiki.apache.org/confluence/display/NIFI/Wormhole+Connections
>>
>> On Tue, Mar 13, 2018 at 10:48 AM, Anil Rai <an...@gmail.com> wrote:
>> > Team,
>> >
>> > My question is around re-use.
>> > Templates : We are creating flows for different use cases. Most of them
>> have
>> > a requirement for retry. Say the API call fails, we would like to retry
>> 3
>> > time after certain interval before giving up. We have created a retry
>> > template that works perfect.
>> > Q: If I have to use this template, then i have to import this template
>> in
>> > all the use case flows. If we have a change to this template, then i
>> would
>> > be forced to go to every flow that uses this template and make that
>> change.
>> > Date, String, Number formatting : I have seen couple of ways of doing
>> this
>> > formatting. Like in UpdateAttribute, Jolt transformation, ExecuteScript.
>> > Q: Like in Java, can we have this utility in one place that can be used
>> by
>> > all my flows at run time by passing the required Parameters?
>> >
>> > Thanks
>> > Anil
>> >
>> >
>>
>
>
Re: Reuse of templates and formatting functions
Posted by Anil Rai <an...@gmail.com>.
Thanks Bryan. That helps.
We were also thinking of exposing standard formatting functions as an API
that other data flows can call and use.
Regards
Anil
On Tue, Mar 13, 2018 at 1:13 PM, Bryan Bende <bb...@gmail.com> wrote:
> Anil,
>
> It sounds like what you are interested in would be some combination of
> the referenceable process groups [1] or worm-hole connections [2],
> which were ideas that were discussed, but haven't been implemented.
>
> You might be able to make your situation slightly better by using the
> NiFi Registry instead of templates...
>
> As an example, you could design a stand-alone process group for your
> retry logic and place it under version control in the registry. Then
> you import this flow from the registry into various points of your
> overall flow.
>
> You could then go back to the original standalone process group and
> make a change to the logic and save version 2 to the registry. At this
> point all the other places you imported it to will show an upgrade and
> you can change their version to v2 to bring in the change.
>
> It still requires manual upgrading of all the versioned process
> groups, but at least they are all tied back to the same version, where
> as templates are not linked to anything once they instantiated.
>
> -Bryan
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/
> Reference-able+Process+Groups
> [2] https://cwiki.apache.org/confluence/display/NIFI/Wormhole+Connections
>
> On Tue, Mar 13, 2018 at 10:48 AM, Anil Rai <an...@gmail.com> wrote:
> > Team,
> >
> > My question is around re-use.
> > Templates : We are creating flows for different use cases. Most of them
> have
> > a requirement for retry. Say the API call fails, we would like to retry 3
> > time after certain interval before giving up. We have created a retry
> > template that works perfect.
> > Q: If I have to use this template, then i have to import this template in
> > all the use case flows. If we have a change to this template, then i
> would
> > be forced to go to every flow that uses this template and make that
> change.
> > Date, String, Number formatting : I have seen couple of ways of doing
> this
> > formatting. Like in UpdateAttribute, Jolt transformation, ExecuteScript.
> > Q: Like in Java, can we have this utility in one place that can be used
> by
> > all my flows at run time by passing the required Parameters?
> >
> > Thanks
> > Anil
> >
> >
>
Re: Reuse of templates and formatting functions
Posted by Bryan Bende <bb...@gmail.com>.
Anil,
It sounds like what you are interested in would be some combination of
the referenceable process groups [1] or worm-hole connections [2],
which were ideas that were discussed, but haven't been implemented.
You might be able to make your situation slightly better by using the
NiFi Registry instead of templates...
As an example, you could design a stand-alone process group for your
retry logic and place it under version control in the registry. Then
you import this flow from the registry into various points of your
overall flow.
You could then go back to the original standalone process group and
make a change to the logic and save version 2 to the registry. At this
point all the other places you imported it to will show an upgrade and
you can change their version to v2 to bring in the change.
It still requires manual upgrading of all the versioned process
groups, but at least they are all tied back to the same version, where
as templates are not linked to anything once they instantiated.
-Bryan
[1] https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups
[2] https://cwiki.apache.org/confluence/display/NIFI/Wormhole+Connections
On Tue, Mar 13, 2018 at 10:48 AM, Anil Rai <an...@gmail.com> wrote:
> Team,
>
> My question is around re-use.
> Templates : We are creating flows for different use cases. Most of them have
> a requirement for retry. Say the API call fails, we would like to retry 3
> time after certain interval before giving up. We have created a retry
> template that works perfect.
> Q: If I have to use this template, then i have to import this template in
> all the use case flows. If we have a change to this template, then i would
> be forced to go to every flow that uses this template and make that change.
> Date, String, Number formatting : I have seen couple of ways of doing this
> formatting. Like in UpdateAttribute, Jolt transformation, ExecuteScript.
> Q: Like in Java, can we have this utility in one place that can be used by
> all my flows at run time by passing the required Parameters?
>
> Thanks
> Anil
>
>