[jira] [Commented] (NIFI-7557) Cache large/common FlowFile attributes when restoring FlowFile Repository

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190182#comment-17190182 ] 

ASF subversion and git services commented on NIFI-7557:
-------------------------------------------------------

Commit fd068fe978da5a4a3e5df14b9a8e43f51e88c606 in nifi's branch refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=fd068fe ]

NIFI-7557: uses a canonical representation of strings when recovering data from FlowFile Repository in order to avoid using huge amounts of heap when not necessary
- Fixed some problems with unit/integration tests

This closes #4507.

Signed-off-by: Bryan Bende <bb...@apache.org>


> Cache large/common FlowFile attributes when restoring FlowFile Repository
> -------------------------------------------------------------------------
>
>                 Key: NIFI-7557
>                 URL: https://issues.apache.org/jira/browse/NIFI-7557
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When NiFi is restarted, it restores FlowFiles from the repository. Each attribute on a FlowFile is read from disk and put into a HashMap. There are times when a Processor will add a large attribute to every FlowFile that it sees, and this results in using much more heap upon NiFi restart to store FlowFiles than it does while NiFi is running. This is because the Processor holds the value of that FlowFile as a single String object and adds that String to the HashMap of attributes on every FlowFile.
> However, on restart, NiFi deserializes a byte stream to come up with the attribute value. As a result, each FlowFile that has that attribute value ends up with its own String object, even though the same value is repeated many times.
> As a result, a huge amount of heap may be used on restart, causing NiFi to encounter OOME when attempting to restore the FlowFile Repository.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)