You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Payne (Jira)" <ji...@apache.org> on 2020/06/06 22:19:00 UTC
[jira] [Commented] (NIFI-7510) MergeRecords not working after
RouteOnAttribute
[ https://issues.apache.org/jira/browse/NIFI-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127441#comment-17127441 ]
Mark Payne commented on NIFI-7510:
----------------------------------
[~garfield69au] this sounds like the correct behavior. When you split the data using SplitRecord, it adds a series of "fragment.*" attributes. One of those is "fragment.count". This is the total number of "splits" being generated. When you use MergeRecord with Defragment mode, it is waiting to receive however many splits are specified by the "fragment.count" attribute. So this works fine without RouteOnAttribute. But when you add in RouteOnAttribute, you are filtering out some of the FlowFiles, so all of the FlowFiles don't arrive at MergeRecord. Thus, it will continue waiting until it receives them all without merging anything together.
I do think we can improve MergeRecord by allowing for a "Bin Expiration Strategy" that is either "Merge Available FlowFiles" or "Fail Available FlowFiles." Right now, if the Max Bin Age is reached, the FlowFiles that belong together go to failure. If we added this new property, it would allow for the user to specify that the Processor should merge together whatever it can and just ignore the rest.
That said, the notion of splitting the data apart and then merging it back together again, especially with Record-based processors, is something that you should avoid if at all possible. Doing that results in far more work for the NiFi engine and provides far worse performance (commonly an order of magnitude difference in performance) than when you keep records together in a single FlowFile.
So your flow could probably look something more like:
GetFile -> ConvertExcelToCSV -> UpdateRecord -> ExecuteSQLRecord -> PutFile
It's not entirely clear to me if you need to explicitly treat records differently if (count = 0) or if the intent is just to filter them out. But you can adapt the flow to handle that, if necessary, by adding "PartitionRecord -> RouteOnAttribute" in between ExecuteSQLRecord and PutFile.
> MergeRecords not working after RouteOnAttribute
> -----------------------------------------------
>
> Key: NIFI-7510
> URL: https://issues.apache.org/jira/browse/NIFI-7510
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.11.4
> Environment: MS Windows 10
> Reporter: Shane Downey
> Priority: Minor
>
> Here's the scenario:
> I need to process an arbitrary length spreadsheet or CSV file.
> For each row in the file, I need to extract a value from a column called "Key" and pass it into a SQL Query. If the query returns 0 rows I need to write the source "Key" into an exception file.
> For the successful queries, I need to write the full output to a spreadsheet file.
>
> Setup:
> GetFile-->ConvertExceltoCSVProcessor-->SplitRecord(CSVReader/JSONRecordSetWriter)->
> ExtractText("Key"/Value)-->UpdateAttribute("arg.1"=value)->
> ExecuteSQLRecord()-->RouteOnAttribute(rowcount=0)->LogAttribute
> RouteOnAttribute(rowcount!=0)-->MergeRecords(Defragment)->PutFile
>
> In the above scenario, the FlowFile blocks in the queue leading into MergeRecords (never actually makes it into MergeRecords).
>
> If I change the merge strategy to Bin then data does flow through to the output file in "chunks" of rows.
> If I remove RouteOnAttribute from the flow, the solution works as expected and my output file is created as expected.
>
> Thus - there appears to be an issue using RouteOnAttribute within a SplitRecord/MergeRecord block and using the defragment merge strategy.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)