GetFile Usage

[KhajaAsmath Mohammed <md...@gmail.com>]

Hi,

I have a question regarding how the getfile runs in the background. Lets
say I have 1000 files in the directory, when I use getfile will all 1000
files are read once and loaded into queue or is it sequentially by file?

My requirement is to load all files in queue at a time so that I can come
up with approach of dependency management.

Thanks,
Asmath

Re: GetFile Usage

[Edward Armes <ed...@gmail.com>]

Just to note, I believe that this is the maximum number of files in a given
window before the processor chooses to yield.

The processor could finish before reaching that limit if told to by the
scheduling framework.

It's worth having a quick look in the source code to see how and when the
processor releases FlowFiles to the queue in relation to the number files
streamed to the content repository.

Edward

On Fri, 22 May 2020, 16:13 Luis, <lc...@openpartner.cl> wrote:

> Hi Asmath,
>
> I have used GetFile several times, and you can control what you mention
> with one of the parameters of the Processor. Check the attached image.
>
> I haven't gone that far like 1.000 files, but I guess it is not a problem
> depending on the size of each file versus the available resources of your
> server.
>
> LC
>
>
>
> El vie, 22-05-2020 a las 03:56 -0500, KhajaAsmath Mohammed escribió:
>
> Hi,
>
> I have a question regarding how the getfile runs in the background. Lets
> say I have 1000 files in the directory, when I use getfile will all 1000
> files are read once and loaded into queue or is it sequentially by file?
>
> My requirement is to load all files in queue at a time so that I can come
> up with approach of dependency management.
>
> Thanks,
> Asmath
>
>

Re: GetFile Usage

[Luis <lc...@openpartner.cl>]

Hi Asmath,
I have used GetFile several times, and you can control what you mention
with one of the parameters of the Processor. Check the attached
image. 
I haven't gone that far like 1.000 files, but I guess it is not a
problem depending on the size of each file versus the available
resources of your server.
LC

El vie, 22-05-2020 a las 03:56 -0500, KhajaAsmath Mohammed escribió:
> Hi,
> I have a question regarding how the getfile runs in the background.
> Lets say I have 1000 files in the directory, when I use getfile will
> all 1000 files are read once and loaded into queue or is it
> sequentially by file?
> 
> My requirement is to load all files in queue at a time so that I can
> come up with approach of dependency management.
> 
> Thanks,
> Asmath