You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Brian Ghigiarelli <br...@gmail.com> on 2015/07/14 01:08:03 UTC
Connecting NiFi Processors without Drawing Lines
Hi all,
Is there a way in NiFi to connect two processors without manually drawing
the line between them (or calling the REST interface to create that
connection) ?
To better describe this particular use case, we have Template A as our main
flow. We can deploy Template A onto our main Instance A, and everything
looks great.
Instance B uses the same template, but has an additional ingress path, so
we have created a smaller Template B to pull in that data. We'd like
Template B to drop-in to NiFi and automatically output to one of the
processors in Template A without having to manually connect them each time
in order to ease the burden of the deployment.
>From a flow layout, it looks like:
*Instance A:*
Ingress A --> Normalize --> Magic
*Instance B:*
Ingress A --|
--> Normalize --> Magic
Ingress B --|
Any suggestions would be greatly appreciated. Thanks!
--
Brian Ghigiarelli
Re: Connecting NiFi Processors without Drawing Lines
Posted by Joe Witt <jo...@gmail.com>.
Corey
You're right but i think Brian was just offering the 'action he'd like
available' to him. We'd still need to translate that action into
'showing' the flow as you're saying. What he is asking as a developer
and what you're wanting as an operations person is achievable. We
always have the UX in mind no matter what.
Thanks
Joe
On Tue, Jul 14, 2015 at 8:28 AM, Corey Flowers <cf...@onyxpoint.com> wrote:
> Reading through this, if a processor could connect to another processor
> without a connection, following that flow from an operational perspective
> would be very hard. You could probably work this through rest posting. If
> your template ended with a poster, and your receiving section had a
> listener I believe that would technically work, but then there are
> inefficiencies in pulling the data out of the repo, just to put it right
> back in.
>
> On Tuesday, July 14, 2015, Joe Witt <jo...@gmail.com> wrote:
>
>> What you asked for here (single action adding a template and it is
>> live and connected) is not currently a feature. We'll need to think
>> through that more but the 'concept' I described seems feasible
>> provided a lot of 'ifs'. Those IFs are:
>> - The template being added contains a single output port
>> - The template being added doesn't require any 'sensitive properties'
>> - The template being added is entirely valid when added
>>
>> I'm wondering if we're not to the root of the use case though
>> honestly. It is possible a different design would be more appropriate
>> than templates for this case. We'll need to talk more on this. Are
>> you able to be more concrete with your example?
>>
>> Thanks
>> Joe
>>
>>
>> On Tue, Jul 14, 2015 at 8:16 AM, Brian Ghigiarelli <brianghig@gmail.com
>> <javascript:;>> wrote:
>> > Is it safe to say, then, that it's not an existing feature? I haven't
>> > tried yet, but was wondering if setting up a remote process group
>> pointing
>> > to localhost would do the trick. Drawback to that is having to reach out
>> > to the network interface.
>> >
>> > On Mon, Jul 13, 2015 at 9:58 PM, Brian Ghigiarelli <brianghig@gmail.com
>> <javascript:;>>
>> > wrote:
>> >
>> >> Yeah, I think that would do the trick!
>> >>
>> >> On Mon, Jul 13, 2015 at 7:21 PM, Joe Witt <joe.witt@gmail.com
>> <javascript:;>> wrote:
>> >>
>> >>> Brian
>> >>>
>> >>> So adding a template to the flow is a single call and connecting its
>> >>> output
>> >>> is a second call. Are you simply looking for a single call that would
>> do
>> >>> both?
>> >>>
>> >>> Just want to make sure i follow the idea.
>> >>>
>> >>> One thing that might make sense is to allow a user to drag a template
>> onto
>> >>> a target component which we would treat as signifying that this is
>> where
>> >>> the output of that group goes. This would be cool so long as there is
>> a
>> >>> single output port of the template being instantiated.
>> >>>
>> >>> Am i in the ballpark?
>> >>> On Jul 13, 2015 7:08 PM, "Brian Ghigiarelli" <brianghig@gmail.com
>> <javascript:;>> wrote:
>> >>>
>> >>> > Hi all,
>> >>> >
>> >>> > Is there a way in NiFi to connect two processors without manually
>> >>> drawing
>> >>> > the line between them (or calling the REST interface to create that
>> >>> > connection) ?
>> >>> >
>> >>> > To better describe this particular use case, we have Template A as
>> our
>> >>> main
>> >>> > flow. We can deploy Template A onto our main Instance A, and
>> everything
>> >>> > looks great.
>> >>> >
>> >>> > Instance B uses the same template, but has an additional ingress
>> path,
>> >>> so
>> >>> > we have created a smaller Template B to pull in that data. We'd like
>> >>> > Template B to drop-in to NiFi and automatically output to one of the
>> >>> > processors in Template A without having to manually connect them each
>> >>> time
>> >>> > in order to ease the burden of the deployment.
>> >>> >
>> >>> > From a flow layout, it looks like:
>> >>> >
>> >>> > *Instance A:*
>> >>> > Ingress A --> Normalize --> Magic
>> >>> >
>> >>> > *Instance B:*
>> >>> > Ingress A --|
>> >>> > --> Normalize --> Magic
>> >>> > Ingress B --|
>> >>> >
>> >>> > Any suggestions would be greatly appreciated. Thanks!
>> >>> >
>> >>> > --
>> >>> > Brian Ghigiarelli
>> >>> >
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Brian Ghigiarelli
>> >> 570-878-9139
>> >>
>> >
>> >
>> >
>> > --
>> > Brian Ghigiarelli
>> > 570-878-9139
>>
>
>
> --
> Corey Flowers
> Vice President, Onyx Point, Inc
> (410) 541-6699
> cflowers@onyxpoint.com
>
> -- This account not approved for unencrypted proprietary information --
Re: Connecting NiFi Processors without Drawing Lines
Posted by Corey Flowers <cf...@onyxpoint.com>.
Reading through this, if a processor could connect to another processor
without a connection, following that flow from an operational perspective
would be very hard. You could probably work this through rest posting. If
your template ended with a poster, and your receiving section had a
listener I believe that would technically work, but then there are
inefficiencies in pulling the data out of the repo, just to put it right
back in.
On Tuesday, July 14, 2015, Joe Witt <jo...@gmail.com> wrote:
> What you asked for here (single action adding a template and it is
> live and connected) is not currently a feature. We'll need to think
> through that more but the 'concept' I described seems feasible
> provided a lot of 'ifs'. Those IFs are:
> - The template being added contains a single output port
> - The template being added doesn't require any 'sensitive properties'
> - The template being added is entirely valid when added
>
> I'm wondering if we're not to the root of the use case though
> honestly. It is possible a different design would be more appropriate
> than templates for this case. We'll need to talk more on this. Are
> you able to be more concrete with your example?
>
> Thanks
> Joe
>
>
> On Tue, Jul 14, 2015 at 8:16 AM, Brian Ghigiarelli <brianghig@gmail.com
> <javascript:;>> wrote:
> > Is it safe to say, then, that it's not an existing feature? I haven't
> > tried yet, but was wondering if setting up a remote process group
> pointing
> > to localhost would do the trick. Drawback to that is having to reach out
> > to the network interface.
> >
> > On Mon, Jul 13, 2015 at 9:58 PM, Brian Ghigiarelli <brianghig@gmail.com
> <javascript:;>>
> > wrote:
> >
> >> Yeah, I think that would do the trick!
> >>
> >> On Mon, Jul 13, 2015 at 7:21 PM, Joe Witt <joe.witt@gmail.com
> <javascript:;>> wrote:
> >>
> >>> Brian
> >>>
> >>> So adding a template to the flow is a single call and connecting its
> >>> output
> >>> is a second call. Are you simply looking for a single call that would
> do
> >>> both?
> >>>
> >>> Just want to make sure i follow the idea.
> >>>
> >>> One thing that might make sense is to allow a user to drag a template
> onto
> >>> a target component which we would treat as signifying that this is
> where
> >>> the output of that group goes. This would be cool so long as there is
> a
> >>> single output port of the template being instantiated.
> >>>
> >>> Am i in the ballpark?
> >>> On Jul 13, 2015 7:08 PM, "Brian Ghigiarelli" <brianghig@gmail.com
> <javascript:;>> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > Is there a way in NiFi to connect two processors without manually
> >>> drawing
> >>> > the line between them (or calling the REST interface to create that
> >>> > connection) ?
> >>> >
> >>> > To better describe this particular use case, we have Template A as
> our
> >>> main
> >>> > flow. We can deploy Template A onto our main Instance A, and
> everything
> >>> > looks great.
> >>> >
> >>> > Instance B uses the same template, but has an additional ingress
> path,
> >>> so
> >>> > we have created a smaller Template B to pull in that data. We'd like
> >>> > Template B to drop-in to NiFi and automatically output to one of the
> >>> > processors in Template A without having to manually connect them each
> >>> time
> >>> > in order to ease the burden of the deployment.
> >>> >
> >>> > From a flow layout, it looks like:
> >>> >
> >>> > *Instance A:*
> >>> > Ingress A --> Normalize --> Magic
> >>> >
> >>> > *Instance B:*
> >>> > Ingress A --|
> >>> > --> Normalize --> Magic
> >>> > Ingress B --|
> >>> >
> >>> > Any suggestions would be greatly appreciated. Thanks!
> >>> >
> >>> > --
> >>> > Brian Ghigiarelli
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Brian Ghigiarelli
> >> 570-878-9139
> >>
> >
> >
> >
> > --
> > Brian Ghigiarelli
> > 570-878-9139
>
--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
cflowers@onyxpoint.com
-- This account not approved for unencrypted proprietary information --
Re: Connecting NiFi Processors without Drawing Lines
Posted by Joe Witt <jo...@gmail.com>.
UX is always tricky - we'll figure something out. It's a good idea
just something we never thought of so requires more thought than most
On Tue, Jul 14, 2015 at 8:43 AM, Brian Ghigiarelli <br...@gmail.com> wrote:
> Aldrin - that would be nice, but the second template / instance is
> effectively controlled by another team and isn't available to us on the
> first instance. Does that help the use case?
>
> Yes, Joe, the UX gets tricky because I think what I'm suggesting is
> effectively a global port that any processor can connect to and/or read
> from (for lack of a better term, a "connection bus").
>
> On Tue, Jul 14, 2015 at 8:31 AM, Aldrin Piri <al...@gmail.com> wrote:
>
>> Drat, I knew I would trip up on that. Instance A is instant drop, Instance
>> B is allow through.
>>
>> On Tue, Jul 14, 2015 at 8:30 AM, Aldrin Piri <al...@gmail.com> wrote:
>>
>> > Maybe not the cleanest implementation, but as a quick solution, how about
>> > a RouteOnAttribute with an expression language that checks for a
>> particular
>> > environment variable. This means you have one template but instance A
>> > results in Ingress B being instantly dropped but Instance A allows it to
>> > pass through. Environment variables could easily be set/adjusted as part
>> > of your continuous deployment
>> >
>> > Your template then is:
>> > Ingress A --|
>> > --> Normalize --> Magic
>> > Ingress B --| Route on Attr --|
>> >
>> > On Tue, Jul 14, 2015 at 8:21 AM, Joe Witt <jo...@gmail.com> wrote:
>> >
>> >> What you asked for here (single action adding a template and it is
>> >> live and connected) is not currently a feature. We'll need to think
>> >> through that more but the 'concept' I described seems feasible
>> >> provided a lot of 'ifs'. Those IFs are:
>> >> - The template being added contains a single output port
>> >> - The template being added doesn't require any 'sensitive properties'
>> >> - The template being added is entirely valid when added
>> >>
>> >> I'm wondering if we're not to the root of the use case though
>> >> honestly. It is possible a different design would be more appropriate
>> >> than templates for this case. We'll need to talk more on this. Are
>> >> you able to be more concrete with your example?
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >>
>> >> On Tue, Jul 14, 2015 at 8:16 AM, Brian Ghigiarelli <brianghig@gmail.com
>> >
>> >> wrote:
>> >> > Is it safe to say, then, that it's not an existing feature? I haven't
>> >> > tried yet, but was wondering if setting up a remote process group
>> >> pointing
>> >> > to localhost would do the trick. Drawback to that is having to reach
>> >> out
>> >> > to the network interface.
>> >> >
>> >> > On Mon, Jul 13, 2015 at 9:58 PM, Brian Ghigiarelli <
>> brianghig@gmail.com
>> >> >
>> >> > wrote:
>> >> >
>> >> >> Yeah, I think that would do the trick!
>> >> >>
>> >> >> On Mon, Jul 13, 2015 at 7:21 PM, Joe Witt <jo...@gmail.com>
>> wrote:
>> >> >>
>> >> >>> Brian
>> >> >>>
>> >> >>> So adding a template to the flow is a single call and connecting its
>> >> >>> output
>> >> >>> is a second call. Are you simply looking for a single call that
>> >> would do
>> >> >>> both?
>> >> >>>
>> >> >>> Just want to make sure i follow the idea.
>> >> >>>
>> >> >>> One thing that might make sense is to allow a user to drag a
>> template
>> >> onto
>> >> >>> a target component which we would treat as signifying that this is
>> >> where
>> >> >>> the output of that group goes. This would be cool so long as there
>> >> is a
>> >> >>> single output port of the template being instantiated.
>> >> >>>
>> >> >>> Am i in the ballpark?
>> >> >>> On Jul 13, 2015 7:08 PM, "Brian Ghigiarelli" <br...@gmail.com>
>> >> wrote:
>> >> >>>
>> >> >>> > Hi all,
>> >> >>> >
>> >> >>> > Is there a way in NiFi to connect two processors without manually
>> >> >>> drawing
>> >> >>> > the line between them (or calling the REST interface to create
>> that
>> >> >>> > connection) ?
>> >> >>> >
>> >> >>> > To better describe this particular use case, we have Template A as
>> >> our
>> >> >>> main
>> >> >>> > flow. We can deploy Template A onto our main Instance A, and
>> >> everything
>> >> >>> > looks great.
>> >> >>> >
>> >> >>> > Instance B uses the same template, but has an additional ingress
>> >> path,
>> >> >>> so
>> >> >>> > we have created a smaller Template B to pull in that data. We'd
>> like
>> >> >>> > Template B to drop-in to NiFi and automatically output to one of
>> the
>> >> >>> > processors in Template A without having to manually connect them
>> >> each
>> >> >>> time
>> >> >>> > in order to ease the burden of the deployment.
>> >> >>> >
>> >> >>> > From a flow layout, it looks like:
>> >> >>> >
>> >> >>> > *Instance A:*
>> >> >>> > Ingress A --> Normalize --> Magic
>> >> >>> >
>> >> >>> > *Instance B:*
>> >> >>> > Ingress A --|
>> >> >>> > --> Normalize --> Magic
>> >> >>> > Ingress B --|
>> >> >>> >
>> >> >>> > Any suggestions would be greatly appreciated. Thanks!
>> >> >>> >
>> >> >>> > --
>> >> >>> > Brian Ghigiarelli
>> >> >>> >
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Brian Ghigiarelli
>> >> >> 570-878-9139
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Brian Ghigiarelli
>> >> > 570-878-9139
>> >>
>> >
>> >
>>
Re: Connecting NiFi Processors without Drawing Lines
Posted by Brian Ghigiarelli <br...@gmail.com>.
Aldrin - that would be nice, but the second template / instance is
effectively controlled by another team and isn't available to us on the
first instance. Does that help the use case?
Yes, Joe, the UX gets tricky because I think what I'm suggesting is
effectively a global port that any processor can connect to and/or read
from (for lack of a better term, a "connection bus").
On Tue, Jul 14, 2015 at 8:31 AM, Aldrin Piri <al...@gmail.com> wrote:
> Drat, I knew I would trip up on that. Instance A is instant drop, Instance
> B is allow through.
>
> On Tue, Jul 14, 2015 at 8:30 AM, Aldrin Piri <al...@gmail.com> wrote:
>
> > Maybe not the cleanest implementation, but as a quick solution, how about
> > a RouteOnAttribute with an expression language that checks for a
> particular
> > environment variable. This means you have one template but instance A
> > results in Ingress B being instantly dropped but Instance A allows it to
> > pass through. Environment variables could easily be set/adjusted as part
> > of your continuous deployment
> >
> > Your template then is:
> > Ingress A --|
> > --> Normalize --> Magic
> > Ingress B --| Route on Attr --|
> >
> > On Tue, Jul 14, 2015 at 8:21 AM, Joe Witt <jo...@gmail.com> wrote:
> >
> >> What you asked for here (single action adding a template and it is
> >> live and connected) is not currently a feature. We'll need to think
> >> through that more but the 'concept' I described seems feasible
> >> provided a lot of 'ifs'. Those IFs are:
> >> - The template being added contains a single output port
> >> - The template being added doesn't require any 'sensitive properties'
> >> - The template being added is entirely valid when added
> >>
> >> I'm wondering if we're not to the root of the use case though
> >> honestly. It is possible a different design would be more appropriate
> >> than templates for this case. We'll need to talk more on this. Are
> >> you able to be more concrete with your example?
> >>
> >> Thanks
> >> Joe
> >>
> >>
> >> On Tue, Jul 14, 2015 at 8:16 AM, Brian Ghigiarelli <brianghig@gmail.com
> >
> >> wrote:
> >> > Is it safe to say, then, that it's not an existing feature? I haven't
> >> > tried yet, but was wondering if setting up a remote process group
> >> pointing
> >> > to localhost would do the trick. Drawback to that is having to reach
> >> out
> >> > to the network interface.
> >> >
> >> > On Mon, Jul 13, 2015 at 9:58 PM, Brian Ghigiarelli <
> brianghig@gmail.com
> >> >
> >> > wrote:
> >> >
> >> >> Yeah, I think that would do the trick!
> >> >>
> >> >> On Mon, Jul 13, 2015 at 7:21 PM, Joe Witt <jo...@gmail.com>
> wrote:
> >> >>
> >> >>> Brian
> >> >>>
> >> >>> So adding a template to the flow is a single call and connecting its
> >> >>> output
> >> >>> is a second call. Are you simply looking for a single call that
> >> would do
> >> >>> both?
> >> >>>
> >> >>> Just want to make sure i follow the idea.
> >> >>>
> >> >>> One thing that might make sense is to allow a user to drag a
> template
> >> onto
> >> >>> a target component which we would treat as signifying that this is
> >> where
> >> >>> the output of that group goes. This would be cool so long as there
> >> is a
> >> >>> single output port of the template being instantiated.
> >> >>>
> >> >>> Am i in the ballpark?
> >> >>> On Jul 13, 2015 7:08 PM, "Brian Ghigiarelli" <br...@gmail.com>
> >> wrote:
> >> >>>
> >> >>> > Hi all,
> >> >>> >
> >> >>> > Is there a way in NiFi to connect two processors without manually
> >> >>> drawing
> >> >>> > the line between them (or calling the REST interface to create
> that
> >> >>> > connection) ?
> >> >>> >
> >> >>> > To better describe this particular use case, we have Template A as
> >> our
> >> >>> main
> >> >>> > flow. We can deploy Template A onto our main Instance A, and
> >> everything
> >> >>> > looks great.
> >> >>> >
> >> >>> > Instance B uses the same template, but has an additional ingress
> >> path,
> >> >>> so
> >> >>> > we have created a smaller Template B to pull in that data. We'd
> like
> >> >>> > Template B to drop-in to NiFi and automatically output to one of
> the
> >> >>> > processors in Template A without having to manually connect them
> >> each
> >> >>> time
> >> >>> > in order to ease the burden of the deployment.
> >> >>> >
> >> >>> > From a flow layout, it looks like:
> >> >>> >
> >> >>> > *Instance A:*
> >> >>> > Ingress A --> Normalize --> Magic
> >> >>> >
> >> >>> > *Instance B:*
> >> >>> > Ingress A --|
> >> >>> > --> Normalize --> Magic
> >> >>> > Ingress B --|
> >> >>> >
> >> >>> > Any suggestions would be greatly appreciated. Thanks!
> >> >>> >
> >> >>> > --
> >> >>> > Brian Ghigiarelli
> >> >>> >
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Brian Ghigiarelli
> >> >> 570-878-9139
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Brian Ghigiarelli
> >> > 570-878-9139
> >>
> >
> >
>
Re: Connecting NiFi Processors without Drawing Lines
Posted by Aldrin Piri <al...@gmail.com>.
Drat, I knew I would trip up on that. Instance A is instant drop, Instance
B is allow through.
On Tue, Jul 14, 2015 at 8:30 AM, Aldrin Piri <al...@gmail.com> wrote:
> Maybe not the cleanest implementation, but as a quick solution, how about
> a RouteOnAttribute with an expression language that checks for a particular
> environment variable. This means you have one template but instance A
> results in Ingress B being instantly dropped but Instance A allows it to
> pass through. Environment variables could easily be set/adjusted as part
> of your continuous deployment
>
> Your template then is:
> Ingress A --|
> --> Normalize --> Magic
> Ingress B --| Route on Attr --|
>
> On Tue, Jul 14, 2015 at 8:21 AM, Joe Witt <jo...@gmail.com> wrote:
>
>> What you asked for here (single action adding a template and it is
>> live and connected) is not currently a feature. We'll need to think
>> through that more but the 'concept' I described seems feasible
>> provided a lot of 'ifs'. Those IFs are:
>> - The template being added contains a single output port
>> - The template being added doesn't require any 'sensitive properties'
>> - The template being added is entirely valid when added
>>
>> I'm wondering if we're not to the root of the use case though
>> honestly. It is possible a different design would be more appropriate
>> than templates for this case. We'll need to talk more on this. Are
>> you able to be more concrete with your example?
>>
>> Thanks
>> Joe
>>
>>
>> On Tue, Jul 14, 2015 at 8:16 AM, Brian Ghigiarelli <br...@gmail.com>
>> wrote:
>> > Is it safe to say, then, that it's not an existing feature? I haven't
>> > tried yet, but was wondering if setting up a remote process group
>> pointing
>> > to localhost would do the trick. Drawback to that is having to reach
>> out
>> > to the network interface.
>> >
>> > On Mon, Jul 13, 2015 at 9:58 PM, Brian Ghigiarelli <brianghig@gmail.com
>> >
>> > wrote:
>> >
>> >> Yeah, I think that would do the trick!
>> >>
>> >> On Mon, Jul 13, 2015 at 7:21 PM, Joe Witt <jo...@gmail.com> wrote:
>> >>
>> >>> Brian
>> >>>
>> >>> So adding a template to the flow is a single call and connecting its
>> >>> output
>> >>> is a second call. Are you simply looking for a single call that
>> would do
>> >>> both?
>> >>>
>> >>> Just want to make sure i follow the idea.
>> >>>
>> >>> One thing that might make sense is to allow a user to drag a template
>> onto
>> >>> a target component which we would treat as signifying that this is
>> where
>> >>> the output of that group goes. This would be cool so long as there
>> is a
>> >>> single output port of the template being instantiated.
>> >>>
>> >>> Am i in the ballpark?
>> >>> On Jul 13, 2015 7:08 PM, "Brian Ghigiarelli" <br...@gmail.com>
>> wrote:
>> >>>
>> >>> > Hi all,
>> >>> >
>> >>> > Is there a way in NiFi to connect two processors without manually
>> >>> drawing
>> >>> > the line between them (or calling the REST interface to create that
>> >>> > connection) ?
>> >>> >
>> >>> > To better describe this particular use case, we have Template A as
>> our
>> >>> main
>> >>> > flow. We can deploy Template A onto our main Instance A, and
>> everything
>> >>> > looks great.
>> >>> >
>> >>> > Instance B uses the same template, but has an additional ingress
>> path,
>> >>> so
>> >>> > we have created a smaller Template B to pull in that data. We'd like
>> >>> > Template B to drop-in to NiFi and automatically output to one of the
>> >>> > processors in Template A without having to manually connect them
>> each
>> >>> time
>> >>> > in order to ease the burden of the deployment.
>> >>> >
>> >>> > From a flow layout, it looks like:
>> >>> >
>> >>> > *Instance A:*
>> >>> > Ingress A --> Normalize --> Magic
>> >>> >
>> >>> > *Instance B:*
>> >>> > Ingress A --|
>> >>> > --> Normalize --> Magic
>> >>> > Ingress B --|
>> >>> >
>> >>> > Any suggestions would be greatly appreciated. Thanks!
>> >>> >
>> >>> > --
>> >>> > Brian Ghigiarelli
>> >>> >
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Brian Ghigiarelli
>> >> 570-878-9139
>> >>
>> >
>> >
>> >
>> > --
>> > Brian Ghigiarelli
>> > 570-878-9139
>>
>
>
Re: Connecting NiFi Processors without Drawing Lines
Posted by Aldrin Piri <al...@gmail.com>.
Maybe not the cleanest implementation, but as a quick solution, how about a
RouteOnAttribute with an expression language that checks for a particular
environment variable. This means you have one template but instance A
results in Ingress B being instantly dropped but Instance A allows it to
pass through. Environment variables could easily be set/adjusted as part
of your continuous deployment
Your template then is:
Ingress A --|
--> Normalize --> Magic
Ingress B --| Route on Attr --|
On Tue, Jul 14, 2015 at 8:21 AM, Joe Witt <jo...@gmail.com> wrote:
> What you asked for here (single action adding a template and it is
> live and connected) is not currently a feature. We'll need to think
> through that more but the 'concept' I described seems feasible
> provided a lot of 'ifs'. Those IFs are:
> - The template being added contains a single output port
> - The template being added doesn't require any 'sensitive properties'
> - The template being added is entirely valid when added
>
> I'm wondering if we're not to the root of the use case though
> honestly. It is possible a different design would be more appropriate
> than templates for this case. We'll need to talk more on this. Are
> you able to be more concrete with your example?
>
> Thanks
> Joe
>
>
> On Tue, Jul 14, 2015 at 8:16 AM, Brian Ghigiarelli <br...@gmail.com>
> wrote:
> > Is it safe to say, then, that it's not an existing feature? I haven't
> > tried yet, but was wondering if setting up a remote process group
> pointing
> > to localhost would do the trick. Drawback to that is having to reach out
> > to the network interface.
> >
> > On Mon, Jul 13, 2015 at 9:58 PM, Brian Ghigiarelli <br...@gmail.com>
> > wrote:
> >
> >> Yeah, I think that would do the trick!
> >>
> >> On Mon, Jul 13, 2015 at 7:21 PM, Joe Witt <jo...@gmail.com> wrote:
> >>
> >>> Brian
> >>>
> >>> So adding a template to the flow is a single call and connecting its
> >>> output
> >>> is a second call. Are you simply looking for a single call that would
> do
> >>> both?
> >>>
> >>> Just want to make sure i follow the idea.
> >>>
> >>> One thing that might make sense is to allow a user to drag a template
> onto
> >>> a target component which we would treat as signifying that this is
> where
> >>> the output of that group goes. This would be cool so long as there is
> a
> >>> single output port of the template being instantiated.
> >>>
> >>> Am i in the ballpark?
> >>> On Jul 13, 2015 7:08 PM, "Brian Ghigiarelli" <br...@gmail.com>
> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > Is there a way in NiFi to connect two processors without manually
> >>> drawing
> >>> > the line between them (or calling the REST interface to create that
> >>> > connection) ?
> >>> >
> >>> > To better describe this particular use case, we have Template A as
> our
> >>> main
> >>> > flow. We can deploy Template A onto our main Instance A, and
> everything
> >>> > looks great.
> >>> >
> >>> > Instance B uses the same template, but has an additional ingress
> path,
> >>> so
> >>> > we have created a smaller Template B to pull in that data. We'd like
> >>> > Template B to drop-in to NiFi and automatically output to one of the
> >>> > processors in Template A without having to manually connect them each
> >>> time
> >>> > in order to ease the burden of the deployment.
> >>> >
> >>> > From a flow layout, it looks like:
> >>> >
> >>> > *Instance A:*
> >>> > Ingress A --> Normalize --> Magic
> >>> >
> >>> > *Instance B:*
> >>> > Ingress A --|
> >>> > --> Normalize --> Magic
> >>> > Ingress B --|
> >>> >
> >>> > Any suggestions would be greatly appreciated. Thanks!
> >>> >
> >>> > --
> >>> > Brian Ghigiarelli
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Brian Ghigiarelli
> >> 570-878-9139
> >>
> >
> >
> >
> > --
> > Brian Ghigiarelli
> > 570-878-9139
>
Re: Connecting NiFi Processors without Drawing Lines
Posted by Joe Witt <jo...@gmail.com>.
What you asked for here (single action adding a template and it is
live and connected) is not currently a feature. We'll need to think
through that more but the 'concept' I described seems feasible
provided a lot of 'ifs'. Those IFs are:
- The template being added contains a single output port
- The template being added doesn't require any 'sensitive properties'
- The template being added is entirely valid when added
I'm wondering if we're not to the root of the use case though
honestly. It is possible a different design would be more appropriate
than templates for this case. We'll need to talk more on this. Are
you able to be more concrete with your example?
Thanks
Joe
On Tue, Jul 14, 2015 at 8:16 AM, Brian Ghigiarelli <br...@gmail.com> wrote:
> Is it safe to say, then, that it's not an existing feature? I haven't
> tried yet, but was wondering if setting up a remote process group pointing
> to localhost would do the trick. Drawback to that is having to reach out
> to the network interface.
>
> On Mon, Jul 13, 2015 at 9:58 PM, Brian Ghigiarelli <br...@gmail.com>
> wrote:
>
>> Yeah, I think that would do the trick!
>>
>> On Mon, Jul 13, 2015 at 7:21 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>>> Brian
>>>
>>> So adding a template to the flow is a single call and connecting its
>>> output
>>> is a second call. Are you simply looking for a single call that would do
>>> both?
>>>
>>> Just want to make sure i follow the idea.
>>>
>>> One thing that might make sense is to allow a user to drag a template onto
>>> a target component which we would treat as signifying that this is where
>>> the output of that group goes. This would be cool so long as there is a
>>> single output port of the template being instantiated.
>>>
>>> Am i in the ballpark?
>>> On Jul 13, 2015 7:08 PM, "Brian Ghigiarelli" <br...@gmail.com> wrote:
>>>
>>> > Hi all,
>>> >
>>> > Is there a way in NiFi to connect two processors without manually
>>> drawing
>>> > the line between them (or calling the REST interface to create that
>>> > connection) ?
>>> >
>>> > To better describe this particular use case, we have Template A as our
>>> main
>>> > flow. We can deploy Template A onto our main Instance A, and everything
>>> > looks great.
>>> >
>>> > Instance B uses the same template, but has an additional ingress path,
>>> so
>>> > we have created a smaller Template B to pull in that data. We'd like
>>> > Template B to drop-in to NiFi and automatically output to one of the
>>> > processors in Template A without having to manually connect them each
>>> time
>>> > in order to ease the burden of the deployment.
>>> >
>>> > From a flow layout, it looks like:
>>> >
>>> > *Instance A:*
>>> > Ingress A --> Normalize --> Magic
>>> >
>>> > *Instance B:*
>>> > Ingress A --|
>>> > --> Normalize --> Magic
>>> > Ingress B --|
>>> >
>>> > Any suggestions would be greatly appreciated. Thanks!
>>> >
>>> > --
>>> > Brian Ghigiarelli
>>> >
>>>
>>
>>
>>
>> --
>> Brian Ghigiarelli
>> 570-878-9139
>>
>
>
>
> --
> Brian Ghigiarelli
> 570-878-9139
Re: Connecting NiFi Processors without Drawing Lines
Posted by Brian Ghigiarelli <br...@gmail.com>.
Is it safe to say, then, that it's not an existing feature? I haven't
tried yet, but was wondering if setting up a remote process group pointing
to localhost would do the trick. Drawback to that is having to reach out
to the network interface.
On Mon, Jul 13, 2015 at 9:58 PM, Brian Ghigiarelli <br...@gmail.com>
wrote:
> Yeah, I think that would do the trick!
>
> On Mon, Jul 13, 2015 at 7:21 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Brian
>>
>> So adding a template to the flow is a single call and connecting its
>> output
>> is a second call. Are you simply looking for a single call that would do
>> both?
>>
>> Just want to make sure i follow the idea.
>>
>> One thing that might make sense is to allow a user to drag a template onto
>> a target component which we would treat as signifying that this is where
>> the output of that group goes. This would be cool so long as there is a
>> single output port of the template being instantiated.
>>
>> Am i in the ballpark?
>> On Jul 13, 2015 7:08 PM, "Brian Ghigiarelli" <br...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > Is there a way in NiFi to connect two processors without manually
>> drawing
>> > the line between them (or calling the REST interface to create that
>> > connection) ?
>> >
>> > To better describe this particular use case, we have Template A as our
>> main
>> > flow. We can deploy Template A onto our main Instance A, and everything
>> > looks great.
>> >
>> > Instance B uses the same template, but has an additional ingress path,
>> so
>> > we have created a smaller Template B to pull in that data. We'd like
>> > Template B to drop-in to NiFi and automatically output to one of the
>> > processors in Template A without having to manually connect them each
>> time
>> > in order to ease the burden of the deployment.
>> >
>> > From a flow layout, it looks like:
>> >
>> > *Instance A:*
>> > Ingress A --> Normalize --> Magic
>> >
>> > *Instance B:*
>> > Ingress A --|
>> > --> Normalize --> Magic
>> > Ingress B --|
>> >
>> > Any suggestions would be greatly appreciated. Thanks!
>> >
>> > --
>> > Brian Ghigiarelli
>> >
>>
>
>
>
> --
> Brian Ghigiarelli
> 570-878-9139
>
--
Brian Ghigiarelli
570-878-9139
Re: Connecting NiFi Processors without Drawing Lines
Posted by Brian Ghigiarelli <br...@gmail.com>.
Yeah, I think that would do the trick!
On Mon, Jul 13, 2015 at 7:21 PM, Joe Witt <jo...@gmail.com> wrote:
> Brian
>
> So adding a template to the flow is a single call and connecting its output
> is a second call. Are you simply looking for a single call that would do
> both?
>
> Just want to make sure i follow the idea.
>
> One thing that might make sense is to allow a user to drag a template onto
> a target component which we would treat as signifying that this is where
> the output of that group goes. This would be cool so long as there is a
> single output port of the template being instantiated.
>
> Am i in the ballpark?
> On Jul 13, 2015 7:08 PM, "Brian Ghigiarelli" <br...@gmail.com> wrote:
>
> > Hi all,
> >
> > Is there a way in NiFi to connect two processors without manually drawing
> > the line between them (or calling the REST interface to create that
> > connection) ?
> >
> > To better describe this particular use case, we have Template A as our
> main
> > flow. We can deploy Template A onto our main Instance A, and everything
> > looks great.
> >
> > Instance B uses the same template, but has an additional ingress path, so
> > we have created a smaller Template B to pull in that data. We'd like
> > Template B to drop-in to NiFi and automatically output to one of the
> > processors in Template A without having to manually connect them each
> time
> > in order to ease the burden of the deployment.
> >
> > From a flow layout, it looks like:
> >
> > *Instance A:*
> > Ingress A --> Normalize --> Magic
> >
> > *Instance B:*
> > Ingress A --|
> > --> Normalize --> Magic
> > Ingress B --|
> >
> > Any suggestions would be greatly appreciated. Thanks!
> >
> > --
> > Brian Ghigiarelli
> >
>
--
Brian Ghigiarelli
570-878-9139
Re: Connecting NiFi Processors without Drawing Lines
Posted by Joe Witt <jo...@gmail.com>.
Brian
So adding a template to the flow is a single call and connecting its output
is a second call. Are you simply looking for a single call that would do
both?
Just want to make sure i follow the idea.
One thing that might make sense is to allow a user to drag a template onto
a target component which we would treat as signifying that this is where
the output of that group goes. This would be cool so long as there is a
single output port of the template being instantiated.
Am i in the ballpark?
On Jul 13, 2015 7:08 PM, "Brian Ghigiarelli" <br...@gmail.com> wrote:
> Hi all,
>
> Is there a way in NiFi to connect two processors without manually drawing
> the line between them (or calling the REST interface to create that
> connection) ?
>
> To better describe this particular use case, we have Template A as our main
> flow. We can deploy Template A onto our main Instance A, and everything
> looks great.
>
> Instance B uses the same template, but has an additional ingress path, so
> we have created a smaller Template B to pull in that data. We'd like
> Template B to drop-in to NiFi and automatically output to one of the
> processors in Template A without having to manually connect them each time
> in order to ease the burden of the deployment.
>
> From a flow layout, it looks like:
>
> *Instance A:*
> Ingress A --> Normalize --> Magic
>
> *Instance B:*
> Ingress A --|
> --> Normalize --> Magic
> Ingress B --|
>
> Any suggestions would be greatly appreciated. Thanks!
>
> --
> Brian Ghigiarelli
>