[jira] [Created] (NIFI-10553) MergeContent Prematurely Evicts Bins

["Eric Secules (Jira)" <ji...@apache.org>]

Eric Secules created NIFI-10553:
-----------------------------------

             Summary: MergeContent Prematurely Evicts Bins
                 Key: NIFI-10553
                 URL: https://issues.apache.org/jira/browse/NIFI-10553
             Project: Apache NiFi
          Issue Type: Bug
          Components: Core Framework
    Affects Versions: 1.16.3, 1.14.0
            Reporter: Eric Secules
             Fix For: 1.18.0


When NiFi's merge processors are configured to defragment, the user wants flowfiles merged in a specific way according to the `fragment.` attributes. Hoever, when MergeDocuments is handling many unique values for `fragment.identifier` it opens up one bin per value until it reaches the `MAX_BIN_COUNT` parameter configured on this processor. This parameter is there to limit memory used by merging too many things all at once. It is not certain that the user will be able to set this to an appropriate value for every flow, and the consequence is that evicting a partially filled bin will result in possible downstream issues and flowfiles stuck in the input connection of MergeDocuments.

 

Instead of this behaviour, the merge processor should penalize and requeue flowfiles that don't fit in any of the existing bins if we have reached the max number of bins already. Penalizing non-matching flowfiles will give time for the ones needed to complete the existing bins to arrive.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)