You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by KhajaAsmath Mohammed <md...@gmail.com> on 2020/04/28 07:07:11 UTC

List file and Ftoetch File - wait until all files are read

Hi,

We are using List file and Fetch file to continuously monitor shared
location to read the files and process the next level of jobs. Issue we are
facing is to wait for all the files to be read before running the next jobs.

Currently list file has 5 files but during the fetch process , it reads
file one by one. Next processor after fetch will work on the files but we
have to wait until all the files are read before the next processor runs.
How to acheive this.

Number of files keeps changing.

Thanks,
Asmath

Re: List file and Ftoetch File - wait until all files are read

Posted by KhajaAsmath Mohammed <md...@gmail.com>.

Thanks Mark. This is really helpful. Will try this option. 

Sent from my iPhone

> On Apr 29, 2020, at 1:00 AM, Mark Payne <ma...@hotmail.com> wrote:
> 
> Asmath,
> 
> NiFi is not job oriented, and it doesn’t really have any notion of being “done” - it monitors the directory continuously, as you noted.
> 
> What might be of use to you is the MonitorActivity processor and the Wait/Notify Processors. I recommend that you read up on their documentation. But you could potentially do something along the lines of:
> 
> 
> ListFile — success —> MonitorActivity — success —> FetchFile —> Wait
>                                                            — inactive —> Notify
> 
> So it should pull the data in, and then wait.
> The MonitorActivity will detect that no more data has come in for some amount of time (say 5 minutes) and as a result determine that the “job is done” and send a FlowFile to Notify to unblock the waiting FlowFiles.
> 
> Hope this helps
> -Mark
> 
> 
>> On Apr 28, 2020, at 3:07 AM, KhajaAsmath Mohammed <md...@gmail.com> wrote:
>> 
>> Hi,
>> 
>> We are using List file and Fetch file to continuously monitor shared location to read the files and process the next level of jobs. Issue we are facing is to wait for all the files to be read before running the next jobs.
>> 
>> Currently list file has 5 files but during the fetch process , it reads file one by one. Next processor after fetch will work on the files but we have to wait until all the files are read before the next processor runs. How to acheive this.
>> 
>> Number of files keeps changing.
>> 
>> Thanks,
>> Asmath
>

Re: List file and Ftoetch File - wait until all files are read

Posted by Mark Payne <ma...@hotmail.com>.

Asmath,

NiFi is not job oriented, and it doesn’t really have any notion of being “done” - it monitors the directory continuously, as you noted.

What might be of use to you is the MonitorActivity processor and the Wait/Notify Processors. I recommend that you read up on their documentation. But you could potentially do something along the lines of:

ListFile — success —> MonitorActivity — success —> FetchFile —> Wait
                                                            — inactive —> Notify

So it should pull the data in, and then wait.
The MonitorActivity will detect that no more data has come in for some amount of time (say 5 minutes) and as a result determine that the “job is done” and send a FlowFile to Notify to unblock the waiting FlowFiles.

Hope this helps
-Mark

> On Apr 28, 2020, at 3:07 AM, KhajaAsmath Mohammed <md...@gmail.com> wrote:
> 
> Hi,
> 
> We are using List file and Fetch file to continuously monitor shared location to read the files and process the next level of jobs. Issue we are facing is to wait for all the files to be read before running the next jobs.
> 
> Currently list file has 5 files but during the fetch process , it reads file one by one. Next processor after fetch will work on the files but we have to wait until all the files are read before the next processor runs. How to acheive this.
> 
> Number of files keeps changing.
> 
> Thanks,
> Asmath