RE: [Non-DoD Source] Re: Communication between flows

["Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US)" <st...@mail.mil>]

Thanks, Bryan!  I will give the MergeContent processor a go and see if it works.

I appreciate your quick reply!

Thank you,

Steven K. Byers 
DXC Technology - Contractor
Software Developer - Joint Medical Logistics Functional Development Center (JMLFDC)
Defense Health Agency (DHA)/ Health Information Technology (HIT) Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC
1681 Nelson Street, Fort Detrick, MD  21702 
(443) 538-7575 | (410) 872-4923
Email:     steven.k.byers.ctr@mail.mil



-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com] 
Sent: Wednesday, July 05, 2017 1:01 PM
To: dev@nifi.apache.org
Subject: [Non-DoD Source] Re: Communication between flows

All active links contained in this email were disabled.  Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser.  




----

Steve,

In 1.2.0 there were some new processors added called Wait/Notify...

With those you could send your original JSON (before splitting) to a Wait processor and tell it to wait until the signal count is equal to the number of splits, then you could put a Notify processor right after PutMongo connected to the success relationship. For example, if
100 JSON documents get split out, the Wait processor is waiting for
100 signals or until it times out, and signals are only sent after successful insertion to Mongo. You can checkout this blog for an example [1].

In 1.1.2, you might be able to put a MergeContent processor configured to run in defragment mode connected to the success relationship of PutMongo. Defragment mode is used to undo the splitting that was done by an unpstream processors, so it will only defragment and merged back together if all the fragments made it through.

-Bryan

[1] Caution-https://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/


On Wed, Jul 5, 2017 at 12:46 PM, Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US) <st...@mail.mil> wrote:
> Is there a mechanism or technique for communicating the results of a flow file to its "sister" flow files?
>
> Here is a high-level description of what I am doing:
>
> Input to my flow is a JSON array of documents that get split (SplitJson) into individual documents and each document becomes a distinct flow file.  Each document (flow file) gets validated against a JSON schema (ValidateJson) then gets updated into a Mongo collection (PutMongoUpdate).  At the end of all this, I want to do some post processing but only if all documents processed successfully.
>
> When a failure occurs (either in the validation or the Mongo update) is there a way to communicate that to the success branch of the flow process so a decision can be made about whether to proceed to post processing or not.
>
> I am using NiFi 1.1.2
>
> Thank you for any guidance you can offer,
>
> Steve
>
>
>