You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Payne (Jira)" <ji...@apache.org> on 2023/05/16 14:43:00 UTC
[jira] [Created] (NIFI-11557) Eliminate use of NIO.2 for any performance-critical parts of application
Mark Payne created NIFI-11557:
---------------------------------
Summary: Eliminate use of NIO.2 for any performance-critical parts of application
Key: NIFI-11557
URL: https://issues.apache.org/jira/browse/NIFI-11557
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework, Extensions
Reporter: Mark Payne
Assignee: Mark Payne
Fix For: 1.latest, 2.latest
The FileSystemRepository (content repo implementation) as well as ListFile both make use of the {{Files.walkFileTree}} method. Recently, I worked with a user who had horribly long startup times. Thread dumps show that the time was almost entirely in the FileSystemRepository's {{initializeRepository}} method as it is walking the file tree in order to determine which archive files can be cleaned up next. This is done during startup and again periodically in background threads.
I made a small modification locally to instead use the standard synchronous IO methods ( {{File.listFiles}} method. I used GenerateFlowFile to generate 1-byte FlowFiles and set {{nifi.content.claim.max.appendable.size=1 B}} in nifi.properties in order to generate a huge number of files - about 1.2 million files in the content repository and restarted a few times. Additionally, added some log lines to show how long this part of the startup process took.
With the existing code, startup took 210 seconds (3.5 mins). With the new implementation, it took 6.7 seconds. The appears to be due to the fact that when using NIO.2 for every file, it does an individual disk access to obtain File attributes, while when using the {{File.listFiles}} method the File objects that are returned already have the necessary attributes. As a result, the NIO.2 approach makes millions of disk accesses that are unnecessary. As the number of files in the repository grows, the discrepancy also grows.
We need to eliminate any use of {{File.walkFileTree}} for any performance-critical parts of the codebase.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)