getFTP repeatedly downloads the same set of files

[Surendra Lalwani <su...@gmail.com>]

Hi

I am using getFTP in my data flow and what I observed is that it repeatedly
downloads the same set of files at each polling interval. What I was
expecting was that once a file is downloaded it will not re-download the
same file again.

Is this correct behavior or I am missing something here.

-Surendra

Re: getFTP repeatedly downloads the same set of files

[Joe Witt <jo...@gmail.com>]

Hello

The GetFTP processor itself doesn't support idempotent behavior.
We've moved to a 'List' and 'Fetch' model for processors where that is
useful as you'll see in ListSFTP and FetchSFTP.  We have not built
that pattern for GetFTP.  It was originally built with the idea that
it would be taking the files out of a directory rather than as a
cloning/archive sort of use case.

Thanks
Joe

On Sun, Dec 13, 2015 at 2:51 PM, Surendra Lalwani
<su...@gmail.com> wrote:
> Hi
>
> I am using getFTP in my data flow and what I observed is that it repeatedly
> downloads the same set of files at each polling interval. What I was
> expecting was that once a file is downloaded it will not re-download the
> same file again.
>
> Is this correct behavior or I am missing something here.
>
> -Surendra
>
>