[jira] [Created] (NIFI-5918) MergeRecord works wrong with Defragment strategy

["Alexander Bukarev (JIRA)" <ji...@apache.org>]

Alexander Bukarev created NIFI-5918:
---------------------------------------

             Summary: MergeRecord works wrong with Defragment strategy
                 Key: NIFI-5918
                 URL: https://issues.apache.org/jira/browse/NIFI-5918
             Project: Apache NiFi
          Issue Type: Bug
          Components: Extensions
    Affects Versions: 1.8.0
            Reporter: Alexander Bukarev


*Steps*
# Create the simple flow: 
#* {{GenerateFlowFile}} (with constant payload "txt1,txt2" and 10 secs schedulling) 
#* -> {{SplitContent}} (with comma as a separator)
#* -> some chain of processors which get "txt1" and "txt2" as a inbound params and produce flowfiles with more than 1 record ((!) that's important). For example, I use {{ExtractText}} (to get "txt1" and "txt2" as an attribute), then {{ExecuteSQLRecord}} (to execute SQL using "txt1" and "txt2" as a parameter)
#* -> {{MergeRecord}} (with *Defragment* merge strategy - (!) that's important)
#* -> {{LogAttribute}} or whatever you prefer to observe the merge result
# Now just run the flow

*Result:* we'll see an error in logs like {panel}Could not merge bin with 1 FlowFiles because of the 'fragment.count' attribute had a value of '2' but only 1 of 2 FlowFiles were encountered before this bin was evicted (due to to Max Bin Age being reached or due to the Maximum Number of Bins being exceeded).{panel}

*Expected result:* the flow file containing records from both SQL queries (for "txt1" and "txt2")


The cause is {{RecordBinManager}} uses {{fragment.count}} flow file attribute to calculate required *record* number to release the bin. However, the attribute contains the number of *flow files* instead. As in above scenario each file contains more than 1 records (at least 2) that means {{RecordBin}} thinks the bin is "full enough" when first flow file arrives (because it contains >= 2 records and {{fragment.count}} is equal to 2 in the scenario). So the bin is released wrongly.

I think there is a mistake and in *Defragment* mode we are interested in a number of flow files and never in records number. In opposite, we should care about a number of records usin Bin-Packaging Algorithm.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)