[DISCUSS] Feature proposal: "Function Groups" and wormholes

[Joe Witt <jo...@gmail.com>]

Come on with a subject like that people must want to read on...

The model of configuration hierarchy nifi supports with Process
Groups/ports today enforces a very explicit connection from component
or group to another component or group.  This is great in that it is
very intentional and makes it easy for operator to understand
precisely how the data flows.  However, this means that a given
process group cannot be easily reused - you have to make copies of it.

What we propose then is to introduce a concept of a 'Function Group'.
It would be similar in structure to a Process Group yet it would have
a very important difference.  These 'Function Groups' would be
reference-able from anywhere in the flow.  They would have zero or one
input ports and they would have zero or more output ports.  Once a
function group is defined then references to it can be placed anywhere
onto the flow it is needed just like dragging a new processor onto the
flow works.  The framework would automatically retain the context of
the where the objects actually came from in the flow and thus where
they need to come back out.

Similar in concept, but for different use cases then is the idea of a
'wormhole' type input port and output port.  These would be virtual
ports whereby you could refer to that port anywhere on the graph that
you need to and the actual port would be somewhere specific.  In this
sense you would not be required to have explicit connections at all
times.

I am guessing this is a little tough to follow with both my wording
and the lack of visuals.  Will turn this into a wiki page with some
screenshots/concept drawings to help communicate this better.
However, wanted to put this out to start finding if there are any
strong opinions on this concept.

Our current model is explicit but can be too rigid and cause people to
create flows that have way more connections and duplication than are
truly necessary or intended.  By offering these implicit and explicit
models with some UX enhancements I believe we can retain the ability
for an operator to understand and follow how the flow works but reduce
the complexity that can grow in a completely explicit approach as we
have today.

Thanks
Joe

Re: [DISCUSS] Feature proposal: "Function Groups" and wormholes

[Joe Witt <jo...@gmail.com>]

Wormhole writeup:
https://cwiki.apache.org/confluence/display/NIFI/Wormhole+Connections

Function Group writeup:
https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups

On Mon, Aug 10, 2015 at 10:41 AM, Joe Witt <jo...@gmail.com> wrote:
> Mike
>
> "Wormholes sounds functionally similar to JMS Queues with topic subscriptions?"
> Yep - you've got it.  Though we'd still make it easy for folks to
> understand the directionality of the flow because we don't want to
> lose that.
>
>
> Function Groups:  Yeah i can't believe we didn't really come up with
> this before now.  It was inspired by the wormhole concept.  These
> would be crazy useful.
>
>
>
> Thanks
> Joe
>
> On Mon, Aug 10, 2015 at 10:36 AM, Mike Drob <md...@apache.org> wrote:
>> Wormholes sounds functionally similar to JMS Queues with topic
>> subscriptions? Function groups are something I've wished for before but
>> didn't think it was possible.
>>
>> On Sun, Aug 9, 2015 at 8:27 AM, Joe Witt <jo...@gmail.com> wrote:
>>
>>> Toivo
>>>
>>> I think from your response you did understand it quite well.  I'll try
>>> to turn this and a series of other features/concepts into a series of
>>> wiki pages so it can be more readily consumed, improved, and commented
>>> on.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Sun, Aug 9, 2015 at 8:04 AM, Toivo Adams <to...@gmail.com> wrote:
>>> > Not sure I understand 'wormhole' concept correctly.
>>> >
>>> > We use a lot of request-response type of processing.
>>> > Processing pipeline contain several processors.
>>> > Processing is done mostly sequentially (and same parts parallel).
>>> > During processing anything can happen. But our goal is to send always
>>> > responds back.
>>> > In case of errors response contains errors data.
>>> >
>>> > There can be different types of problems; some are warnings, some
>>> notices,
>>> > some errors and some fatal errors.
>>> >
>>> > In case of fatal errors we stop processing automatically and jump
>>> directly
>>> > to so called ‘error flow’. (subflow)
>>> > Error flow responsibility is to create response which contains errors and
>>> > send response back to requestor.
>>> > Error flow may also contain some special purpose processors – collect
>>> some
>>> > data for report, alert monitoring, etc.
>>> >
>>> > Now 'wormhole' is very helpful. It’s annoying to add explicit
>>> connections to
>>> > all business processors. Also this reduces readability.
>>> > For us flow should readable and understandable from business point of
>>> view.
>>> > And low level technical behavior should be handled ‘behind the scenes’.
>>> > Ideally business processors should not have ‘redirect to error flow’
>>> output
>>> > (relationship) at all.
>>> >
>>> > At the same time I think 'wormhole' should be used very carefully. Only
>>> when
>>> > really needed.
>>> > Otherwise it may be source of weird bugs and it is hard to follow what is
>>> > going on.
>>> >
>>> >
>>> > Thanks
>>> > toivo
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > View this message in context:
>>> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/DISCUSS-Feature-proposal-Function-Groups-and-wormholes-tp2381p2391.html
>>> > Sent from the Apache NiFi (incubating) Developer List mailing list
>>> archive at Nabble.com.
>>>

Re: [DISCUSS] Feature proposal: "Function Groups" and wormholes

[Joe Witt <jo...@gmail.com>]

Mike

"Wormholes sounds functionally similar to JMS Queues with topic subscriptions?"
Yep - you've got it.  Though we'd still make it easy for folks to
understand the directionality of the flow because we don't want to
lose that.


Function Groups:  Yeah i can't believe we didn't really come up with
this before now.  It was inspired by the wormhole concept.  These
would be crazy useful.



Thanks
Joe

On Mon, Aug 10, 2015 at 10:36 AM, Mike Drob <md...@apache.org> wrote:
> Wormholes sounds functionally similar to JMS Queues with topic
> subscriptions? Function groups are something I've wished for before but
> didn't think it was possible.
>
> On Sun, Aug 9, 2015 at 8:27 AM, Joe Witt <jo...@gmail.com> wrote:
>
>> Toivo
>>
>> I think from your response you did understand it quite well.  I'll try
>> to turn this and a series of other features/concepts into a series of
>> wiki pages so it can be more readily consumed, improved, and commented
>> on.
>>
>> Thanks
>> Joe
>>
>> On Sun, Aug 9, 2015 at 8:04 AM, Toivo Adams <to...@gmail.com> wrote:
>> > Not sure I understand 'wormhole' concept correctly.
>> >
>> > We use a lot of request-response type of processing.
>> > Processing pipeline contain several processors.
>> > Processing is done mostly sequentially (and same parts parallel).
>> > During processing anything can happen. But our goal is to send always
>> > responds back.
>> > In case of errors response contains errors data.
>> >
>> > There can be different types of problems; some are warnings, some
>> notices,
>> > some errors and some fatal errors.
>> >
>> > In case of fatal errors we stop processing automatically and jump
>> directly
>> > to so called ‘error flow’. (subflow)
>> > Error flow responsibility is to create response which contains errors and
>> > send response back to requestor.
>> > Error flow may also contain some special purpose processors – collect
>> some
>> > data for report, alert monitoring, etc.
>> >
>> > Now 'wormhole' is very helpful. It’s annoying to add explicit
>> connections to
>> > all business processors. Also this reduces readability.
>> > For us flow should readable and understandable from business point of
>> view.
>> > And low level technical behavior should be handled ‘behind the scenes’.
>> > Ideally business processors should not have ‘redirect to error flow’
>> output
>> > (relationship) at all.
>> >
>> > At the same time I think 'wormhole' should be used very carefully. Only
>> when
>> > really needed.
>> > Otherwise it may be source of weird bugs and it is hard to follow what is
>> > going on.
>> >
>> >
>> > Thanks
>> > toivo
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/DISCUSS-Feature-proposal-Function-Groups-and-wormholes-tp2381p2391.html
>> > Sent from the Apache NiFi (incubating) Developer List mailing list
>> archive at Nabble.com.
>>

Re: [DISCUSS] Feature proposal: "Function Groups" and wormholes

[Mike Drob <md...@apache.org>]

Wormholes sounds functionally similar to JMS Queues with topic
subscriptions? Function groups are something I've wished for before but
didn't think it was possible.

On Sun, Aug 9, 2015 at 8:27 AM, Joe Witt <jo...@gmail.com> wrote:

> Toivo
>
> I think from your response you did understand it quite well.  I'll try
> to turn this and a series of other features/concepts into a series of
> wiki pages so it can be more readily consumed, improved, and commented
> on.
>
> Thanks
> Joe
>
> On Sun, Aug 9, 2015 at 8:04 AM, Toivo Adams <to...@gmail.com> wrote:
> > Not sure I understand 'wormhole' concept correctly.
> >
> > We use a lot of request-response type of processing.
> > Processing pipeline contain several processors.
> > Processing is done mostly sequentially (and same parts parallel).
> > During processing anything can happen. But our goal is to send always
> > responds back.
> > In case of errors response contains errors data.
> >
> > There can be different types of problems; some are warnings, some
> notices,
> > some errors and some fatal errors.
> >
> > In case of fatal errors we stop processing automatically and jump
> directly
> > to so called ‘error flow’. (subflow)
> > Error flow responsibility is to create response which contains errors and
> > send response back to requestor.
> > Error flow may also contain some special purpose processors – collect
> some
> > data for report, alert monitoring, etc.
> >
> > Now 'wormhole' is very helpful. It’s annoying to add explicit
> connections to
> > all business processors. Also this reduces readability.
> > For us flow should readable and understandable from business point of
> view.
> > And low level technical behavior should be handled ‘behind the scenes’.
> > Ideally business processors should not have ‘redirect to error flow’
> output
> > (relationship) at all.
> >
> > At the same time I think 'wormhole' should be used very carefully. Only
> when
> > really needed.
> > Otherwise it may be source of weird bugs and it is hard to follow what is
> > going on.
> >
> >
> > Thanks
> > toivo
> >
> >
> >
> >
> > --
> > View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/DISCUSS-Feature-proposal-Function-Groups-and-wormholes-tp2381p2391.html
> > Sent from the Apache NiFi (incubating) Developer List mailing list
> archive at Nabble.com.
>

Re: [DISCUSS] Feature proposal: "Function Groups" and wormholes

[Joe Witt <jo...@gmail.com>]

Toivo

I think from your response you did understand it quite well.  I'll try
to turn this and a series of other features/concepts into a series of
wiki pages so it can be more readily consumed, improved, and commented
on.

Thanks
Joe

On Sun, Aug 9, 2015 at 8:04 AM, Toivo Adams <to...@gmail.com> wrote:
> Not sure I understand 'wormhole' concept correctly.
>
> We use a lot of request-response type of processing.
> Processing pipeline contain several processors.
> Processing is done mostly sequentially (and same parts parallel).
> During processing anything can happen. But our goal is to send always
> responds back.
> In case of errors response contains errors data.
>
> There can be different types of problems; some are warnings, some notices,
> some errors and some fatal errors.
>
> In case of fatal errors we stop processing automatically and jump directly
> to so called ‘error flow’. (subflow)
> Error flow responsibility is to create response which contains errors and
> send response back to requestor.
> Error flow may also contain some special purpose processors – collect some
> data for report, alert monitoring, etc.
>
> Now 'wormhole' is very helpful. It’s annoying to add explicit connections to
> all business processors. Also this reduces readability.
> For us flow should readable and understandable from business point of view.
> And low level technical behavior should be handled ‘behind the scenes’.
> Ideally business processors should not have ‘redirect to error flow’ output
> (relationship) at all.
>
> At the same time I think 'wormhole' should be used very carefully. Only when
> really needed.
> Otherwise it may be source of weird bugs and it is hard to follow what is
> going on.
>
>
> Thanks
> toivo
>
>
>
>
> --
> View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/DISCUSS-Feature-proposal-Function-Groups-and-wormholes-tp2381p2391.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.

Re: [DISCUSS] Feature proposal: "Function Groups" and wormholes

[Toivo Adams <to...@gmail.com>]

Not sure I understand 'wormhole' concept correctly.

We use a lot of request-response type of processing.
Processing pipeline contain several processors.
Processing is done mostly sequentially (and same parts parallel).
During processing anything can happen. But our goal is to send always
responds back.
In case of errors response contains errors data.

There can be different types of problems; some are warnings, some notices,
some errors and some fatal errors.

In case of fatal errors we stop processing automatically and jump directly
to so called ‘error flow’. (subflow)
Error flow responsibility is to create response which contains errors and
send response back to requestor.
Error flow may also contain some special purpose processors – collect some
data for report, alert monitoring, etc.

Now 'wormhole' is very helpful. It’s annoying to add explicit connections to
all business processors. Also this reduces readability.
For us flow should readable and understandable from business point of view.
And low level technical behavior should be handled ‘behind the scenes’.
Ideally business processors should not have ‘redirect to error flow’ output
(relationship) at all. 

At the same time I think 'wormhole' should be used very carefully. Only when
really needed.
Otherwise it may be source of weird bugs and it is hard to follow what is
going on.


Thanks
toivo




--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/DISCUSS-Feature-proposal-Function-Groups-and-wormholes-tp2381p2391.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.