ListSFTP is hanging

[David Marrow <dm...@cloudgrasp.com>]

We have been using Nifi for over a year and we just turned up a new cluster.  
We move around 6TB a day of small to large files.   We are having an issue
of the ListSFTP missing files.   I know this can happen if a file with an
older date is moved into the directory because the lister is maintaining
state.   However it also seems to hang when there are 10k plus files.   I am
running Nifi 1.6 on Ubuntu 18.  The cluster has plenty of memory, CPU, and
disk space.   I am also using the distributed cache because we haven't
migrated to 1.8 yet.   

We have 20 different data flows all with their own logic.  We connect the
Lister to a remote port that is connected to a remote process group and then
distributed across the cluster to a FetchSFTP that deletes the files after
they are loaded.  

We move files into the input directory so we have permission to delete them
from the Nifi Fetch.  We are doing a find which orders the files to make
sure that we don't grab old files.  This could still be an issue and cause
us to miss a few files but it still doesn't explain why when the lister is
running and there are files to pull nothing gets pulled. 

Any suggestion for idea would be appreciated.  

Dave



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: ListSFTP is hanging

[Koji Kawamura <ij...@gmail.com>]

Hello Dave,

Although you already mentioned that you haven't migrated to 1.8 yet, I
recommend to do so because:
- 1.8 adds new 'Listing Strategy' property to ListSFTP processor,
which may help your use-case to not miss any files to list
https://issues.apache.org/jira/browse/NIFI-5406
- 1.8 also introduced new 'Load Balancing' capability at connections,
which simplifies List, Distribute, Fetch pattern
https://issues.apache.org/jira/browse/NIFI-5516

It's all about file timestamps and the state tracked by ListSFTP. If
you can share the ListSFTP state and the files and their last modified
timestamps those are expected to be fetched (but not), we can find how
they are missed. But the reason why we had to implement NIFI-5406 was,
handling every case by only timestamps was not possible. Even if we
find a reason, it doesn't guarantee that we can find a workaround
without updating NiFi to 1.8.

Thanks,
Koji
On Sat, Nov 10, 2018 at 7:55 AM David Marrow <dm...@cloudgrasp.com> wrote:
>
> We have been using Nifi for over a year and we just turned up a new cluster.
> We move around 6TB a day of small to large files.   We are having an issue
> of the ListSFTP missing files.   I know this can happen if a file with an
> older date is moved into the directory because the lister is maintaining
> state.   However it also seems to hang when there are 10k plus files.   I am
> running Nifi 1.6 on Ubuntu 18.  The cluster has plenty of memory, CPU, and
> disk space.   I am also using the distributed cache because we haven't
> migrated to 1.8 yet.
>
> We have 20 different data flows all with their own logic.  We connect the
> Lister to a remote port that is connected to a remote process group and then
> distributed across the cluster to a FetchSFTP that deletes the files after
> they are loaded.
>
> We move files into the input directory so we have permission to delete them
> from the Nifi Fetch.  We are doing a find which orders the files to make
> sure that we don't grab old files.  This could still be an issue and cause
> us to miss a few files but it still doesn't explain why when the lister is
> running and there are files to pull nothing gets pulled.
>
> Any suggestion for idea would be appreciated.
>
> Dave
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/