You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Manish Gupta 8 <mg...@sapient.com> on 2017/11/18 06:32:25 UTC

Wait for N Files before starting to process

Hi,

I am working on building a flow in NiFi that involve cloning a file to 2 or more processing flows, and then wait (on some processor) for all the parallel flows to finish (in parallel) and execute a another flow.

Is there any processor in Nifi which can do this out of the box i.e. wait for N file in input queue and then start its processing and timeout if expected N files didn't arrive in some time T? If not, what's a good way of achieving this.

I have such a flow implemented in Akka, and want to migrate it to NiFi. I am using NiFi 1.3



Thanks,

Manish Gupta
Senior Specialist Platform | AI & Data Engineering Practice
________________________________

"Oxygen", Tower C, Ground - 3rd floor,
Plot No. 7, Sector 144 Expressway, Noida, UP, India

Mobile: +91 981 059 1361
Office: +91 (120) 479 5000  Ext : 75398
Email: mgupta50@sapient.com<ma...@sapient.com>
sapientconsulting.com<https://www.sapientconsulting.com/>




Re: Wait for N Files before starting to process

Posted by Jeff <jt...@gmail.com>.
Manish,

You're welcome, and thanks to Koji for writing up that blog!

On Mon, Nov 20, 2017 at 9:27 AM Manish Gupta 8 <mg...@sapient.com> wrote:

> Thank Jeff. This is exactly what I was searching for. Thank You.
>
>
>
>
>
> *From:* Jeff [mailto:jtswork@gmail.com]
> *Sent:* Monday, November 20, 2017 7:41 PM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: Wait for N Files before starting to process
>
>
>
> Hello Manish,
>
>
>
> I was answering a dev-list question and provided a URL to Koji's blog
> about Wait/Notify processors [1].  Please take a look and feel free to ask
> questions about how you you can integrate the Wait/Notify processors into
> your flow.
>
>
>
> [1]
> http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/#alternative-solution-waitnotify
>
>
>
> On Sat, Nov 18, 2017 at 8:28 AM Manish Gupta 8 <mg...@sapient.com>
> wrote:
>
> Thanks Andy. I will search for that thread.
>
>
>
> *From:* Andy Loughran [mailto:andylockran@gmail.com]
> *Sent:* Saturday, November 18, 2017 2:53 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Wait for N Files before starting to process
>
>
>
> Yo can chain together the wait and notify - there’s a message on this list
> for how to do that from about 4 months ago. - I asked the same question :)
>
>
>
> Andy
>
>
>
>
>
> Sent from my iPhone
>
>
> On 18 Nov 2017, at 06:32, Manish Gupta 8 <mg...@sapient.com> wrote:
>
> Hi,
>
>
>
> I am working on building a flow in NiFi that involve cloning a file to 2
> or more processing flows, and then wait (on some processor) for all the
> parallel flows to finish (in parallel) and execute a another flow.
>
>
>
> Is there any processor in Nifi which can do this out of the box i.e. wait
> for N file in input queue and then start its processing and timeout if
> expected N files didn’t arrive in some time T? If not, what’s a good way of
> achieving this.
>
>
>
> I have such a flow implemented in Akka, and want to migrate it to NiFi. I
> am using NiFi 1.3
>
>
>
> <Picture (Device Independent Bitmap) 1.jpg>
>
>
>
> Thanks,
>
>
>
> *Manish Gupta*
>
> *Senior Specialist Platform | AI & Data Engineering Practice*
>
> *____**__**_**_**________**_______**____**_____*
>
>
>
> “Oxygen”, Tower C, Ground - 3rd floor,
>
> Plot No. 7, Sector 144 Expressway, Noida, UP, India
>
>
>
> *Mobile:* +91 981 059 1361 <+91%2098105%2091361>
>
> *Office:* +91 (120) 479 5000 <+91%20120%20479%205000>  *Ext : *75398
>
> *Email:* mgupta50@sapient.com
>
> *sapientconsulting.com* <https://www.sapientconsulting.com/>
>
>
>
>
>
>
>
>

RE: Wait for N Files before starting to process

Posted by Manish Gupta 8 <mg...@sapient.com>.
Thank Jeff. This is exactly what I was searching for. Thank You.


From: Jeff [mailto:jtswork@gmail.com]
Sent: Monday, November 20, 2017 7:41 PM
To: users@nifi.apache.org
Subject: Re: Wait for N Files before starting to process

Hello Manish,

I was answering a dev-list question and provided a URL to Koji's blog about Wait/Notify processors [1].  Please take a look and feel free to ask questions about how you you can integrate the Wait/Notify processors into your flow.

[1] http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/#alternative-solution-waitnotify

On Sat, Nov 18, 2017 at 8:28 AM Manish Gupta 8 <mg...@sapient.com>> wrote:
Thanks Andy. I will search for that thread.

From: Andy Loughran [mailto:andylockran@gmail.com<ma...@gmail.com>]
Sent: Saturday, November 18, 2017 2:53 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Wait for N Files before starting to process

Yo can chain together the wait and notify - there’s a message on this list for how to do that from about 4 months ago. - I asked the same question :)

Andy


Sent from my iPhone

On 18 Nov 2017, at 06:32, Manish Gupta 8 <mg...@sapient.com>> wrote:
Hi,

I am working on building a flow in NiFi that involve cloning a file to 2 or more processing flows, and then wait (on some processor) for all the parallel flows to finish (in parallel) and execute a another flow.

Is there any processor in Nifi which can do this out of the box i.e. wait for N file in input queue and then start its processing and timeout if expected N files didn’t arrive in some time T? If not, what’s a good way of achieving this.

I have such a flow implemented in Akka, and want to migrate it to NiFi. I am using NiFi 1.3

<Picture (Device Independent Bitmap) 1.jpg>

Thanks,

Manish Gupta
Senior Specialist Platform | AI & Data Engineering Practice
________________________________

“Oxygen”, Tower C, Ground - 3rd floor,
Plot No. 7, Sector 144 Expressway, Noida, UP, India

Mobile: +91 981 059 1361<tel:+91%2098105%2091361>
Office: +91 (120) 479 5000<tel:+91%20120%20479%205000>  Ext : 75398
Email: mgupta50@sapient.com<ma...@sapient.com>
sapientconsulting.com<https://www.sapientconsulting.com/>




Re: Wait for N Files before starting to process

Posted by Jeff <jt...@gmail.com>.
Hello Manish,

I was answering a dev-list question and provided a URL to Koji's blog about
Wait/Notify processors [1].  Please take a look and feel free to ask
questions about how you you can integrate the Wait/Notify processors into
your flow.

[1]
http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/#alternative-solution-waitnotify

On Sat, Nov 18, 2017 at 8:28 AM Manish Gupta 8 <mg...@sapient.com> wrote:

> Thanks Andy. I will search for that thread.
>
>
>
> *From:* Andy Loughran [mailto:andylockran@gmail.com]
> *Sent:* Saturday, November 18, 2017 2:53 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Wait for N Files before starting to process
>
>
>
> Yo can chain together the wait and notify - there’s a message on this list
> for how to do that from about 4 months ago. - I asked the same question :)
>
>
>
> Andy
>
>
>
>
>
> Sent from my iPhone
>
>
> On 18 Nov 2017, at 06:32, Manish Gupta 8 <mg...@sapient.com> wrote:
>
> Hi,
>
>
>
> I am working on building a flow in NiFi that involve cloning a file to 2
> or more processing flows, and then wait (on some processor) for all the
> parallel flows to finish (in parallel) and execute a another flow.
>
>
>
> Is there any processor in Nifi which can do this out of the box i.e. wait
> for N file in input queue and then start its processing and timeout if
> expected N files didn’t arrive in some time T? If not, what’s a good way of
> achieving this.
>
>
>
> I have such a flow implemented in Akka, and want to migrate it to NiFi. I
> am using NiFi 1.3
>
>
>
> <Picture (Device Independent Bitmap) 1.jpg>
>
>
>
> Thanks,
>
>
>
> *Manish Gupta*
>
> *Senior Specialist Platform | AI & Data Engineering Practice*
>
> *____**__**_**_**________**_______**____**_____*
>
>
>
> “Oxygen”, Tower C, Ground - 3rd floor,
>
> Plot No. 7, Sector 144 Expressway, Noida, UP, India
>
>
>
> *Mobile:* +91 981 059 1361 <+91%2098105%2091361>
>
> *Office:* +91 (120) 479 5000 <+91%20120%20479%205000>  *Ext : *75398
>
> *Email:* mgupta50@sapient.com
>
> *sapientconsulting.com* <https://www.sapientconsulting.com/>
>
>
>
>
>
>
>
>

RE: Wait for N Files before starting to process

Posted by Manish Gupta 8 <mg...@sapient.com>.
Thanks Andy. I will search for that thread.

From: Andy Loughran [mailto:andylockran@gmail.com]
Sent: Saturday, November 18, 2017 2:53 PM
To: users@nifi.apache.org
Subject: Re: Wait for N Files before starting to process

Yo can chain together the wait and notify - there’s a message on this list for how to do that from about 4 months ago. - I asked the same question :)

Andy


Sent from my iPhone

On 18 Nov 2017, at 06:32, Manish Gupta 8 <mg...@sapient.com>> wrote:
Hi,

I am working on building a flow in NiFi that involve cloning a file to 2 or more processing flows, and then wait (on some processor) for all the parallel flows to finish (in parallel) and execute a another flow.

Is there any processor in Nifi which can do this out of the box i.e. wait for N file in input queue and then start its processing and timeout if expected N files didn’t arrive in some time T? If not, what’s a good way of achieving this.

I have such a flow implemented in Akka, and want to migrate it to NiFi. I am using NiFi 1.3

<Picture (Device Independent Bitmap) 1.jpg>

Thanks,

Manish Gupta
Senior Specialist Platform | AI & Data Engineering Practice
________________________________

“Oxygen”, Tower C, Ground - 3rd floor,
Plot No. 7, Sector 144 Expressway, Noida, UP, India

Mobile: +91 981 059 1361
Office: +91 (120) 479 5000  Ext : 75398
Email: mgupta50@sapient.com<ma...@sapient.com>
sapientconsulting.com<https://www.sapientconsulting.com/>




Re: Wait for N Files before starting to process

Posted by Andy Loughran <an...@gmail.com>.
Yo can chain together the wait and notify - there’s a message on this list for how to do that from about 4 months ago. - I asked the same question :)

Andy



Sent from my iPhone

> On 18 Nov 2017, at 06:32, Manish Gupta 8 <mg...@sapient.com> wrote:
> 
> Hi,
>  
> I am working on building a flow in NiFi that involve cloning a file to 2 or more processing flows, and then wait (on some processor) for all the parallel flows to finish (in parallel) and execute a another flow.
>  
> Is there any processor in Nifi which can do this out of the box i.e. wait for N file in input queue and then start its processing and timeout if expected N files didn’t arrive in some time T? If not, what’s a good way of achieving this.
>  
> I have such a flow implemented in Akka, and want to migrate it to NiFi. I am using NiFi 1.3
>  
> <Picture (Device Independent Bitmap) 1.jpg>
>  
> Thanks,
>  
> Manish Gupta
> Senior Specialist Platform | AI & Data Engineering Practice
> ________________________________
>  
> “Oxygen”, Tower C, Ground - 3rd floor,
> Plot No. 7, Sector 144 Expressway, Noida, UP, India
>  
> Mobile: +91 981 059 1361
> Office: +91 (120) 479 5000  Ext : 75398
> Email: mgupta50@sapient.com
> sapientconsulting.com
>  
>  
>