You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Tomislav Novosel <to...@gmail.com> on 2019/01/28 08:11:17 UTC
Minimum file age
Hi all,
I'm having issue with SFTPList processor in Nifi. When reading files from
folder where another process writes files, it lists the same file multiple
time and ingests file multiple times because modification date of file
changes rapidly as another process writes to it.
It appears that Nifi lists faster than external process writes, so before
the end of writing (conversion of file from one format to another), Nifi
lists file multiple time and then creates duplicates.
There is no property of Minimum file age like in ListFile processor.
How can I resolve this to wait for a moment when the file is converted
completely and then to list file and pass it to FetchSFTP processor?
Thanks in advance,
Tom.
Re: Minimum file age
Posted by Tomislav Novosel <to...@gmail.com>.
Thank you for replies and suggestions.
Yeah, it is a little disturbing lack of feature, but it's ok. For now, as I
have control over remote server, I use dotted file and remove dot (".")
after file is finished.
Ignore dotted file property set to true.
Cheers,
Tom
On Mon, 28 Jan 2019 at 11:09, <Jo...@swisscom.com> wrote:
> Hi Tom
>
>
>
> I suggest to use a Wait Processor (to delay the fetch) together with
> DetectDuplicate Processor. In that way you will fetch the file only once
> and after it has been written completely (as long as you know how long it
> takes in max. to finish writing). I know it’s not nice but that’s how we do
> it for the moment… I’m waiting for this feature as well :-(.
>
>
>
> Cheers Josef
>
>
>
>
>
> *From: *Arpad Boda <ab...@hortonworks.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Monday, 28 January 2019 at 10:17
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Minimum file age
>
>
>
> Hi,
>
>
>
> It’s on the way: https://issues.apache.org/jira/browse/NIFI-5977 :)
>
>
>
> Regards,
>
> Arpad
>
>
>
> *From: *Tomislav Novosel <to...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Monday, 28 January 2019 at 09:19
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Minimum file age
>
>
>
> Hi all,
>
>
>
> I'm having issue with SFTPList processor in Nifi. When reading files from
> folder where another process writes files, it lists the same file multiple
> time and ingests file multiple times because modification date of file
> changes rapidly as another process writes to it.
>
>
>
> It appears that Nifi lists faster than external process writes, so before
> the end of writing (conversion of file from one format to another), Nifi
> lists file multiple time and then creates duplicates.
>
>
>
> There is no property of Minimum file age like in ListFile processor.
>
>
>
> How can I resolve this to wait for a moment when the file is converted
> completely and then to list file and pass it to FetchSFTP processor?
>
>
>
> Thanks in advance,
>
> Tom.
>
Re: Minimum file age
Posted by Arpad Boda <ab...@hortonworks.com>.
Gents,
The fix for this is merged, will be part of 1.10.
Regards,
Arpad
From: Tomislav Novosel <to...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, 31 January 2019 at 12:29
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Minimum file age
Hi all,
@Josef, what do you mean with Wait and DetectDuplicate processors? How to delay fetching by time of conversion?
How can Wait processor know that file is converted completely? If the file is listed again and DetectDuplicate processor
caches identifier, Wair processor will pass flowfile downstream. What in case if file is that big that will be listed three times or four times?
Regards,
Tom
On Mon, 28 Jan 2019 at 11:09, <Jo...@swisscom.com>> wrote:
Hi Tom
I suggest to use a Wait Processor (to delay the fetch) together with DetectDuplicate Processor. In that way you will fetch the file only once and after it has been written completely (as long as you know how long it takes in max. to finish writing). I know it’s not nice but that’s how we do it for the moment… I’m waiting for this feature as well :-(.
Cheers Josef
From: Arpad Boda <ab...@hortonworks.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, 28 January 2019 at 10:17
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Minimum file age
Hi,
It’s on the way: https://issues.apache.org/jira/browse/NIFI-5977 :)
Regards,
Arpad
From: Tomislav Novosel <to...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, 28 January 2019 at 09:19
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Minimum file age
Hi all,
I'm having issue with SFTPList processor in Nifi. When reading files from folder where another process writes files, it lists the same file multiple time and ingests file multiple times because modification date of file changes rapidly as another process writes to it.
It appears that Nifi lists faster than external process writes, so before the end of writing (conversion of file from one format to another), Nifi lists file multiple time and then creates duplicates.
There is no property of Minimum file age like in ListFile processor.
How can I resolve this to wait for a moment when the file is converted completely and then to list file and pass it to FetchSFTP processor?
Thanks in advance,
Tom.
Re: Minimum file age
Posted by Jo...@swisscom.com.
Hi Tom
In the wait processor you can define how long you wanna delay the fetch and that value needs to be the longest expected writing time. Of course this works only if you know what the max. writing time is. In my case this is clear as in max. every hour at least one new file gets created. Sorry for not giving you a better reply.
Cheers Josef
From: Tomislav Novosel <to...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, 31 January 2019 at 12:29
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Minimum file age
Hi all,
@Josef, what do you mean with Wait and DetectDuplicate processors? How to delay fetching by time of conversion?
How can Wait processor know that file is converted completely? If the file is listed again and DetectDuplicate processor
caches identifier, Wair processor will pass flowfile downstream. What in case if file is that big that will be listed three times or four times?
Regards,
Tom
On Mon, 28 Jan 2019 at 11:09, <Jo...@swisscom.com>> wrote:
Hi Tom
I suggest to use a Wait Processor (to delay the fetch) together with DetectDuplicate Processor. In that way you will fetch the file only once and after it has been written completely (as long as you know how long it takes in max. to finish writing). I know it’s not nice but that’s how we do it for the moment… I’m waiting for this feature as well :-(.
Cheers Josef
From: Arpad Boda <ab...@hortonworks.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, 28 January 2019 at 10:17
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Minimum file age
Hi,
It’s on the way: https://issues.apache.org/jira/browse/NIFI-5977 :)
Regards,
Arpad
From: Tomislav Novosel <to...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, 28 January 2019 at 09:19
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Minimum file age
Hi all,
I'm having issue with SFTPList processor in Nifi. When reading files from folder where another process writes files, it lists the same file multiple time and ingests file multiple times because modification date of file changes rapidly as another process writes to it.
It appears that Nifi lists faster than external process writes, so before the end of writing (conversion of file from one format to another), Nifi lists file multiple time and then creates duplicates.
There is no property of Minimum file age like in ListFile processor.
How can I resolve this to wait for a moment when the file is converted completely and then to list file and pass it to FetchSFTP processor?
Thanks in advance,
Tom.
Re: Minimum file age
Posted by Tomislav Novosel <to...@gmail.com>.
Hi all,
@Josef, what do you mean with Wait and DetectDuplicate processors? How to
delay fetching by time of conversion?
How can Wait processor know that file is converted completely? If the file
is listed again and DetectDuplicate processor
caches identifier, Wair processor will pass flowfile downstream. What in
case if file is that big that will be listed three times or four times?
Regards,
Tom
On Mon, 28 Jan 2019 at 11:09, <Jo...@swisscom.com> wrote:
> Hi Tom
>
>
>
> I suggest to use a Wait Processor (to delay the fetch) together with
> DetectDuplicate Processor. In that way you will fetch the file only once
> and after it has been written completely (as long as you know how long it
> takes in max. to finish writing). I know it’s not nice but that’s how we do
> it for the moment… I’m waiting for this feature as well :-(.
>
>
>
> Cheers Josef
>
>
>
>
>
> *From: *Arpad Boda <ab...@hortonworks.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Monday, 28 January 2019 at 10:17
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Minimum file age
>
>
>
> Hi,
>
>
>
> It’s on the way: https://issues.apache.org/jira/browse/NIFI-5977 :)
>
>
>
> Regards,
>
> Arpad
>
>
>
> *From: *Tomislav Novosel <to...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Monday, 28 January 2019 at 09:19
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Minimum file age
>
>
>
> Hi all,
>
>
>
> I'm having issue with SFTPList processor in Nifi. When reading files from
> folder where another process writes files, it lists the same file multiple
> time and ingests file multiple times because modification date of file
> changes rapidly as another process writes to it.
>
>
>
> It appears that Nifi lists faster than external process writes, so before
> the end of writing (conversion of file from one format to another), Nifi
> lists file multiple time and then creates duplicates.
>
>
>
> There is no property of Minimum file age like in ListFile processor.
>
>
>
> How can I resolve this to wait for a moment when the file is converted
> completely and then to list file and pass it to FetchSFTP processor?
>
>
>
> Thanks in advance,
>
> Tom.
>
Re: Minimum file age
Posted by Jo...@swisscom.com.
Hi Tom
I suggest to use a Wait Processor (to delay the fetch) together with DetectDuplicate Processor. In that way you will fetch the file only once and after it has been written completely (as long as you know how long it takes in max. to finish writing). I know it’s not nice but that’s how we do it for the moment… I’m waiting for this feature as well :-(.
Cheers Josef
From: Arpad Boda <ab...@hortonworks.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, 28 January 2019 at 10:17
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Minimum file age
Hi,
It’s on the way: https://issues.apache.org/jira/browse/NIFI-5977 :)
Regards,
Arpad
From: Tomislav Novosel <to...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, 28 January 2019 at 09:19
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Minimum file age
Hi all,
I'm having issue with SFTPList processor in Nifi. When reading files from folder where another process writes files, it lists the same file multiple time and ingests file multiple times because modification date of file changes rapidly as another process writes to it.
It appears that Nifi lists faster than external process writes, so before the end of writing (conversion of file from one format to another), Nifi lists file multiple time and then creates duplicates.
There is no property of Minimum file age like in ListFile processor.
How can I resolve this to wait for a moment when the file is converted completely and then to list file and pass it to FetchSFTP processor?
Thanks in advance,
Tom.
Re: Minimum file age
Posted by Arpad Boda <ab...@hortonworks.com>.
Hi,
It’s on the way: https://issues.apache.org/jira/browse/NIFI-5977 :)
Regards,
Arpad
From: Tomislav Novosel <to...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, 28 January 2019 at 09:19
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Minimum file age
Hi all,
I'm having issue with SFTPList processor in Nifi. When reading files from folder where another process writes files, it lists the same file multiple time and ingests file multiple times because modification date of file changes rapidly as another process writes to it.
It appears that Nifi lists faster than external process writes, so before the end of writing (conversion of file from one format to another), Nifi lists file multiple time and then creates duplicates.
There is no property of Minimum file age like in ListFile processor.
How can I resolve this to wait for a moment when the file is converted completely and then to list file and pass it to FetchSFTP processor?
Thanks in advance,
Tom.