You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Vdsa <vi...@persistent.co.in> on 2017/05/09 11:20:04 UTC
Running processors simulaneously
I have a custom processor which generates some files, listFile takes those
files and moves those files to HDFS, when I run a processor group it starts
executing all the processors and as list file is independent of any input it
starts fetching file and it does not get that. Thus i get error that file
does not exist. How do I wait processor till the time i get files?
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Running-processors-simulaneously-tp15774.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
RE: Running processors simulaneously
Posted by Peter Ho <pe...@live.com>.
One potential option is to generate the files as hidden, and rename the file not to be hidden when the file is finished, so it can be picked up.
Thanks
Peter
From: Matt Burgess<ma...@apache.org>
Sent: Tuesday, May 9, 2017 10:46 AM
To: dev@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Running processors simulaneously
It sounds like your custom processor is generating files on disk, does
it output any flow files, perhaps one flow file per generated on-disk
file? If for each file generated on disk, the custom processor could
output a flow file with the "absolute.path" and "filename" attributes
set, that kind of emulates a ListFile processor, so after your custom
processor you could go directly to a FetchFile (or FetchHDFS or
whatever) processor. That keeps the flow serial rather than ListFile
running concurrently with your processor generating files. If I've
misunderstood anything please let me know.
Regards,
Matt
On Tue, May 9, 2017 at 7:20 AM, Vdsa <vi...@persistent.co.in> wrote:
> I have a custom processor which generates some files, listFile takes those
> files and moves those files to HDFS, when I run a processor group it starts
> executing all the processors and as list file is independent of any input it
> starts fetching file and it does not get that. Thus i get error that file
> does not exist. How do I wait processor till the time i get files?
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Running-processors-simulaneously-tp15774.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Running processors simulaneously
Posted by Vdsa <vi...@persistent.co.in>.
I have implemented solution of creation flow files and working perfectly only
problem I am facing is flow files are transferred after completion of
processor or on commit of session. Is there any way i send flow files as it
is generated to the next processor if i commit session after a
flowfile.Transfer() it gives IllegalStateException.
How can i transfer one one file or in batch to next processor.
Thanks.
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Running-processors-simulaneously-tp15774p15872.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Running processors simulaneously
Posted by Vdsa <vi...@persistent.co.in>.
Thanks matt for the reply, this solution seems interesting, right now only
files are generated on disk, is it possible to generate flow file for each
file? if yes how can i achieve that.
Thanks in advance.
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Running-processors-simulaneously-tp15774p15784.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Running processors simulaneously
Posted by Matt Burgess <ma...@apache.org>.
It sounds like your custom processor is generating files on disk, does
it output any flow files, perhaps one flow file per generated on-disk
file? If for each file generated on disk, the custom processor could
output a flow file with the "absolute.path" and "filename" attributes
set, that kind of emulates a ListFile processor, so after your custom
processor you could go directly to a FetchFile (or FetchHDFS or
whatever) processor. That keeps the flow serial rather than ListFile
running concurrently with your processor generating files. If I've
misunderstood anything please let me know.
Regards,
Matt
On Tue, May 9, 2017 at 7:20 AM, Vdsa <vi...@persistent.co.in> wrote:
> I have a custom processor which generates some files, listFile takes those
> files and moves those files to HDFS, when I run a processor group it starts
> executing all the processors and as list file is independent of any input it
> starts fetching file and it does not get that. Thus i get error that file
> does not exist. How do I wait processor till the time i get files?
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Running-processors-simulaneously-tp15774.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.