[jira] [Created] (NIFI-5178) Stackable content

["eric twilegar (JIRA)" <ji...@apache.org>]

eric twilegar created NIFI-5178:
-----------------------------------

             Summary: Stackable content
                 Key: NIFI-5178
                 URL: https://issues.apache.org/jira/browse/NIFI-5178
             Project: Apache NiFi
          Issue Type: Improvement
            Reporter: eric twilegar


Having an issue where I need to make decision as I'm processing a list of records.

Similar to an upsert/merge type of flow. I need to first check that a record hasn't already been imported or was already imported in some other mechanism prior to routing.

To do this I add an ExecuteSQL to the flow. All I really need is the execute.sql.rowcount:equals(0) statement and the actual results of the executeSQL are useless to me. Simply trying to make a decision on how to branch with RouteOnAttribute.

The work around now is to store the content of the original in an attribute, and then use ReplaceText to then plop it back on after ExecuteSQL processor. This can get quite cumbersome if you have 4 or 5 decision points.

What would be nice is if all processors could instead of replacing content, could PUSH content. Then add a processor called "PopContent" which would just remove the last content that was pushed onto the flowfile. 

If content was a stack then you could go off and get some data, do a few stages with it, then add attributes, and then pop back to the original content. In my case ExecuteSQL wouldn't overwrite content, but instead just push new data onto the stack.

Not sure if LookupService is a better mechanism for this going forward. It's possible I could do a lookup and instead of enriching the data add a boolean type key to be used as a decision point later. Such as "alreadyExistsInDatabase" : "true|false" instead of something like "Store_name" : "Greatest store on earth" that enrichment generally does. I'm sure SQLLookupService is coming.

Adding "original" transfer for executesql might also solve this issue without a lot major refactoring in nifi.

I may put in a ticket for adding original to executeSQL possibly looking at the code myself.

Thanks for the great tool!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)