[jira] [Created] (NIFI-10404) TailFile processor's persistent state not cleaned up

["Zsihovszki Krisztina (Jira)" <ji...@apache.org>]

Zsihovszki Krisztina created NIFI-10404:
-------------------------------------------

             Summary: TailFile processor's persistent state not cleaned up
                 Key: NIFI-10404
                 URL: https://issues.apache.org/jira/browse/NIFI-10404
             Project: Apache NiFi
          Issue Type: Bug
            Reporter: Zsihovszki Krisztina


TailFile processor's persistent state map is not maintained properly, entries are added to this map but the references to non existent files are not moved from the map.

At NiFi restart the states map (storing the TailFileObjects) is empty and it is [filled|https://github.com/hortonworks/nifi/blob/CFM-2.0.4.0/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java#L360] with entries based on the persistent state map. 
Each TailFileObject contains a byte buffer with the size of 65536 bytes. Since the persistent state map is not cleaned up and it may contain thousands of file references, a lot of TailFileObject may be created at startup, each consuming 65536 bytes which can end up in out of memory error. 



The persistent state map needs to be actualized based on the actual files as it is done for the [states map|https://github.com/hortonworks/nifi/blob/CFM-2.0.4.0/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java#L374].

The allocated byte buffer size (65536 bytes) should be configurable to be able to use smaller buffer size (default value would be 65536 bytes for backward compatibility).

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)