[jira] [Updated] (NIFI-6599) MergeRecord fails if 'fragment.count' attribute equals the number of records within a FlowFile where it should wait for remaining FlowFiles

["Koji Kawamura (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Koji Kawamura updated NIFI-6599:
--------------------------------
    Attachment: NIFI-6599.xml

> MergeRecord fails if 'fragment.count' attribute equals the number of records within a FlowFile where it should wait for remaining FlowFiles
> -------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-6599
>                 URL: https://issues.apache.org/jira/browse/NIFI-6599
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Extensions
>            Reporter: Koji Kawamura
>            Assignee: Koji Kawamura
>            Priority: Major
>         Attachments: NIFI-6599.xml
>
>
> RecordBinManager.createThresholds has following code block:
> {code:java}
> if (MergeRecord.MERGE_STRATEGY_DEFRAGMENT.getValue().equals(mergeStrategy)) {
>     fragmentCountAttribute = MergeContent.FRAGMENT_COUNT_ATTRIBUTE;
>     if (!StringUtils.isEmpty(flowfile.getAttribute(fragmentCountAttribute))) {
>         minRecords = Integer.parseInt(flowfile.getAttribute(fragmentCountAttribute));
>     }
> } else {
>     fragmentCountAttribute = null;
> }
> {code}
> The code uses 'fragment.count' as the minRecords. This is wrong because 'fragment.count' represents the number of fragments, i.e. number of FlowFiles holding partial record set.
> This causes a FlowFile to be sent 'failure' relationship where it should be hold in the incoming connection. For example, when a FlowFile is split into two FlowFiles, and each has 2 records in it, 'fragment.count' will be 2. In this case, MergeContent thinks the minRecords is 2, where 4 is correct. Then the first FlowFile is processed, while the 2nd one hasn't arrived, MergeContent misunderstood that the bin reached to the minimum number of records. But since there's only one FlowFile, it sends the FlowFile to 'failure'.
> The issue can be reproduced by the attached template.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)