[jira] [Resolved] (NIFI-13896) Improving TailFile performance

["David Handermann (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Handermann resolved NIFI-13896.
-------------------------------------
    Fix Version/s: 2.0.0
       Resolution: Fixed

> Improving TailFile performance
> ------------------------------
>
>                 Key: NIFI-13896
>                 URL: https://issues.apache.org/jira/browse/NIFI-13896
>             Project: Apache NiFi
>          Issue Type: Improvement
>            Reporter: Lehel Boér
>            Assignee: Lehel Boér
>            Priority: Major
>             Fix For: 2.0.0
>
>         Attachments: tailfile_test.png
>
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> In case of tailing numerous files, the processor is slow because it repeatedly loops over a large number of tailed files and performs several expensive operations.
>  * In the {{OnTrigger}} method, a loop (loop 1) iterates over all tailed files in the state object.
>  * Inside this loop, for each tailed file, the {{recoverRolledFiles}} method is called (loop 2), which then leads to {{consumeFilesFully}} and finally triggers {{{}cleanup{}}}.
>  * In the {{cleanup}} method, another loop (loop 3) iterates over all tailed files in the state again.
>  * During the {{{}cleanup{}}}, {{persistState}} is invoked, which removes any legacy state variables from the NiFi state. These legacy state variables originate from NiFi 1.0, when support for "Multiple Tailed Files" was not available, so state keys didn’t have the "file.x." prefix. As the {{cleanup}} iterates over and persists each tailed file's state, the overall state size grows (adding six entries per tailed file). This increase causes the legacy cleanup loop to become progressively slower with each iteration as the number of state entries grows.
> This can lead to hours of execution time.
>  
> Suggestion for improvement:
>  
>  * Moving out the loop that removes old state entries from cleanup. The cleanup of old entries should be run on the startup instead.
> {code:java}
> for(String key : oldState.toMap().keySet()) {
>     // These states are stored by older version of NiFi, and won't be used anymore.
>     // New states have 'file.<index>.' prefix.
>     if (TailFileState.StateKeys.CHECKSUM.equals(key)
>             || TailFileState.StateKeys.FILENAME.equals(key)
>             || TailFileState.StateKeys.POSITION.equals(key)
>             || TailFileState.StateKeys.TIMESTAMP.equals(key)) {
>         getLogger().info("Removed state {}={} stored by older version of NiFi.", new Object[]{key, oldState.get(key)});
>         continue;
>     }
>     updatedState.put(key, oldState.get(key));
> } {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)