[jira] [Resolved] (NIFI-7454) Directory handle leak in ListFile

["Simon Bence (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Simon Bence resolved NIFI-7454.
-------------------------------
    Resolution: Duplicate

*Summary:*

This issue is fixed by [#NIFI-7292]. There is an ongoing discussion about a 1.12.0 release or a 1.11.5 release to happen soon.

*In details:*

I could reproduce this easily (thank you for the flow!) using MacOS. It turned out that the stream usage keeps a reference alive after the exception (based on what I did see, most probably it's a _sun.nio.fs.UnixPath_, but I did not follow up with this). The fix with the current version of _ListFile_ looks fairly simple:

 
{code:java}
 finally {
    performanceTracker.completeActiveDirectory();
    listing.close();
}
{code}
 

If the listing is closed [here|https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFile.java#L579] the handler will be released. This issue does not effect the master branch however, as the listing of the files was replaced but a different implementation, not using the stream in this way. It is also tested that, with the current head, no similar issue appears.

> Directory handle leak in ListFile
> ---------------------------------
>
>                 Key: NIFI-7454
>                 URL: https://issues.apache.org/jira/browse/NIFI-7454
>             Project: Apache NiFi
>          Issue Type: Bug
>    Affects Versions: 1.11.4
>            Reporter: Paul Kelly
>            Assignee: Simon Bence
>            Priority: Major
>         Attachments: flow.xml.gz
>
>
> ListFile does not seem to be closing directory handles if an exception is raised during the directory walk.  For example, if ListFile is recursively polling a folder and the NiFi user does not have permission to list the contents of one of the subfolders, an "Access Denied" bulletin is raised and the number of DIR handles opened by the NiFi process as shown by lsof increases.  Eventually, NiFi stops working due to "Too many open files".
> This has been tested with NiFi 1.11.4 on Ubuntu 18.04 with Java 11.  I'm not sure in which version this behavior started.
> Attached is a sample flow.xml.gz that demonstrates this behavior.  A clean copy of NiFi 1.11.4 was installed with no modifications to any configuration file.  NiFi is running as the user "xubuntu".  The permissions on the folders are as follows:
> drwxrwxr-x 3 xubuntu xubuntu 4096 May 15 08:55 /home/xubuntu/test
> drwxrwx--- 2 root root 4096 May 15 08:55 /home/xubuntu/test/subtest
> NiFi has permission to list the contents of "/home/xubuntu/test", but not "/home/xubuntu/test/subtest".
> The number of open DIR handles is being monitored with the command:
> lsof +E -P -p 347813 | tr -s " " | cut -d" " -f5 | sort | uniq -c
> where 347813 is the pid of the NiFi process.
> DIR handle counts start at 4 immediately after NiFi is launched and steadily grow to tens of thousands, until file handles are exhausted.  Additionally, these handles do not seem to be released even if the ListFile processor is stopped.
> Please let me know if there is any additional information I can provide.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)