How to wait for multiple processors' flowfile?

[尹文才 <ba...@gmail.com>]

Hi guys, I'm currently using NiFi to do some ETL work and I need to handle
the following problem:

for example I have 3 self-defined processor A, B and C.
all of them contain logic to incrementally update a database table. the
table in C is created by table in A join table in B, so C depends on the
flowfiles from A and B.

So my question is is there a way for C to wait for both flowfiles from A
and B
to arrive and then start the work in C? Thanks.

Re: How to wait for multiple processors' flowfile?

[Bryan Bende <bb...@gmail.com>]

Hello,

The Wait processor has a property "Target Signal Count" which can be a
number or expression language, and tells it how many signals to wait for.

-Bryan


On Wed, May 10, 2017 at 9:58 PM, 尹文才 <ba...@gmail.com> wrote:

> Hi Andrew, thanks for the link, I looked into the wait/notify mechanism
> yesterday and found that one wait processor actually could only wait for
> exactly one notify signal, but my case is like this:
> Processor A, B and C， C has to wait for both flow files from A and B
> conditionally,
> if A and B all outputs flow files, C needs to wait for both flow files and
> use both flow files to update data in C;
> if either A or B doesn't emit flowfile, C also needs to use all incoming
> flow files to update data
> I could know in advance in up-stream processors whether A and B would
> produce flowfiles, but I don't know if it's feasible with NIFI to implement
> the following:
> 1. use wait/notify to wait for multiple different signals(do I have to use
> multiple wait processor chained in a flow to do it?)
> 2. use wait/notify to conditionally wait for a signal
>
> Thanks,
>
> Ben
>
> 2017-05-10 11:49 GMT+08:00 Andrew Psaltis <ps...@gmail.com>:
>
>> Hi,
>> I think the new Wait / Notify processors, can be used to solve this
>> problem. Koji has a great blog post on it [1].
>>
>>
>> [1] http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/
>>
>> Thanks,
>> Andrew
>>
>> On Tue, May 9, 2017 at 11:43 PM, 尹文才 <ba...@gmail.com> wrote:
>>
>>> Hi guys, I'm currently using NiFi to do some ETL work and I need to
>>> handle the following problem:
>>>
>>> for example I have 3 self-defined processor A, B and C.
>>> all of them contain logic to incrementally update a database table. the
>>> table in C is created by table in A join table in B, so C depends on the
>>> flowfiles from A and B.
>>>
>>> So my question is is there a way for C to wait for both flowfiles from A
>>> and B
>>> to arrive and then start the work in C? Thanks.
>>>
>>
>>
>>
>> --
>> Thanks,
>> Andrew
>>
>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>
>
>

Re: How to wait for multiple processors' flowfile?

[尹文才 <ba...@gmail.com>]

Hi Andrew, thanks for the link, I looked into the wait/notify mechanism
yesterday and found that one wait processor actually could only wait for
exactly one notify signal, but my case is like this:
Processor A, B and C， C has to wait for both flow files from A and B
conditionally,
if A and B all outputs flow files, C needs to wait for both flow files and
use both flow files to update data in C;
if either A or B doesn't emit flowfile, C also needs to use all incoming
flow files to update data
I could know in advance in up-stream processors whether A and B would
produce flowfiles, but I don't know if it's feasible with NIFI to implement
the following:
1. use wait/notify to wait for multiple different signals(do I have to use
multiple wait processor chained in a flow to do it?)
2. use wait/notify to conditionally wait for a signal

Thanks,

Ben

2017-05-10 11:49 GMT+08:00 Andrew Psaltis <ps...@gmail.com>:

> Hi,
> I think the new Wait / Notify processors, can be used to solve this
> problem. Koji has a great blog post on it [1].
>
>
> [1] http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/
>
> Thanks,
> Andrew
>
> On Tue, May 9, 2017 at 11:43 PM, 尹文才 <ba...@gmail.com> wrote:
>
>> Hi guys, I'm currently using NiFi to do some ETL work and I need to
>> handle the following problem:
>>
>> for example I have 3 self-defined processor A, B and C.
>> all of them contain logic to incrementally update a database table. the
>> table in C is created by table in A join table in B, so C depends on the
>> flowfiles from A and B.
>>
>> So my question is is there a way for C to wait for both flowfiles from A
>> and B
>> to arrive and then start the work in C? Thanks.
>>
>
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>

Re: How to wait for multiple processors' flowfile?

[Andrew Psaltis <ps...@gmail.com>]

Hi,
I think the new Wait / Notify processors, can be used to solve this
problem. Koji has a great blog post on it [1].


[1] http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/

Thanks,
Andrew

On Tue, May 9, 2017 at 11:43 PM, 尹文才 <ba...@gmail.com> wrote:

> Hi guys, I'm currently using NiFi to do some ETL work and I need to handle
> the following problem:
>
> for example I have 3 self-defined processor A, B and C.
> all of them contain logic to incrementally update a database table. the
> table in C is created by table in A join table in B, so C depends on the
> flowfiles from A and B.
>
> So my question is is there a way for C to wait for both flowfiles from A
> and B
> to arrive and then start the work in C? Thanks.
>



-- 
Thanks,
Andrew

Subscribe to my book: Streaming Data <http://manning.com/psaltis>
<https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>