You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jo...@swisscom.com on 2019/01/24 16:12:47 UTC

ListSFTP Question

Hi guys

We need your advice,… we use the ListSFTP processor to read files on a remote folder. The files gets written like that:


  *   File1
  *   File2
  *   File3 (at the time of “ls” command this file is growing, we don’t know how big it gets or when writing is finished)

So the application on that remote folder writes the files one by one, always one file grows until a certain size (not predictable). So our issue is that NiFi reads the same file multiple time as it grows and the modify timestamp changes. But as we don’t know when writing is finished we don’t know how to drop all “incomplete” ListSFTP flowfiles of the “File3” in the example.

Any help in this, in theory, pretty simple case? The best solution would be to tell the ListSFTP that it should ignore files with a timestamp of lastmodified from at least about x minutes. But there is no such option. We need somehow a ListSFTP delay and we shouldn’t read too new files. But we have no idea how to do that.

Greetings Josef

Re: ListSFTP Question

Posted by Jo...@swisscom.com.
Hi Joe

Done! https://issues.apache.org/jira/browse/NIFI-5977

Cheers Josef


From: Joe Witt <jo...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, 24 January 2019 at 17:18
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: ListSFTP Question

hey josef.  yeah we need to add a min file age property to ListSftp.  please file a jira.

thanks

On Thu, Jan 24, 2019, 11:13 AM <Jo...@swisscom.com> wrote:
Hi guys

We need your advice,… we use the ListSFTP processor to read files on a remote folder. The files gets written like that:


  *   File1
  *   File2
  *   File3 (at the time of “ls” command this file is growing, we don’t know how big it gets or when writing is finished)

So the application on that remote folder writes the files one by one, always one file grows until a certain size (not predictable). So our issue is that NiFi reads the same file multiple time as it grows and the modify timestamp changes. But as we don’t know when writing is finished we don’t know how to drop all “incomplete” ListSFTP flowfiles of the “File3” in the example.

Any help in this, in theory, pretty simple case? The best solution would be to tell the ListSFTP that it should ignore files with a timestamp of lastmodified from at least about x minutes. But there is no such option. We need somehow a ListSFTP delay and we shouldn’t read too new files. But we have no idea how to do that.

Greetings Josef

Re: ListSFTP Question

Posted by Joe Witt <jo...@gmail.com>.
hey josef.  yeah we need to add a min file age property to ListSftp.
please file a jira.

thanks

On Thu, Jan 24, 2019, 11:13 AM <Josef.Zahner1@swisscom.com wrote:

> Hi guys
>
>
>
> We need your advice,… we use the ListSFTP processor to read files on a
> remote folder. The files gets written like that:
>
>
>
>    - File1
>    - File2
>    - File3 (at the time of “ls” command this file is growing, we don’t
>    know how big it gets or when writing is finished)
>
>
>
> So the application on that remote folder writes the files one by one,
> always one file grows until a certain size (not predictable). So our issue
> is that NiFi reads the same file multiple time as it grows and the modify
> timestamp changes. But as we don’t know when writing is finished we don’t
> know how to drop all “incomplete” ListSFTP flowfiles of the “File3” in the
> example.
>
>
>
> Any help in this, in theory, pretty simple case? The best solution would
> be to tell the ListSFTP that it should ignore files with a timestamp of
> lastmodified from at least about x minutes. But there is no such option. We
> need somehow a ListSFTP delay and we shouldn’t read too new files. But we
> have no idea how to do that.
>
>
>
> Greetings Josef
>