NiFi Flow Usage questions

[Eric <ed...@gmail.com>]

Do you have an example workflow of a REST API or external program that
reads the flowfile contents and update attributes with the output?

Can you have processors for child flows impact original flow?
-Compressed file is input into system
-Unpack file
-Run all children files through external program
-Make decision on original based on children  attributes

Last and not least-
Is there a way to export and import attributes between different Nifi
instances? You could have uuid.attributes and uuid.file  to match them on
the distance end.

Thoughts?

Thank you,
Eric

RE: NiFi Flow Usage questions

[Mark Payne <ma...@hotmail.com>]

Hi Eric,

The REST API provides a great deal of power for manipulating flows, by creating/updating/deleting components
and for obtaining information such as metrics about the components on the graph.

The REST API doesn't provide any ability, though, to access FlowFiles from outside of NiFi. FlowFile access (content
and attributes) is intended to be done only through the Processors that are added to the flow.

Currently, the capability to impact a FlowFile based on other FlowFiles (e.g., children) isn't present.
However, this is something that is being worked on ticket NIFI-190 [1]. Once this ticket is complete,
you should be able to "hold" a FlowFile while you process its children, and then once the children
have completed processing, the flow can be notified to release the original FlowFile, optionally passing
some attributes to the original, so that you can make routing decisions, etc.

There are certainly mechanisms for packaging FlowFiles with their attributes and contents. If you are
transferring the data directly from one NiFi to another, this can be accomplished by using site-to-site
to transfer the data between instances. Unfortunately, it looks like the site-to-site information is
not documented much at all in the User Guide, so I have created a ticket for that, as well [2]. This is
really the preferred method for transferring data between two NiFi instances, as it will automatically
detect clusters and perform load balancing. Alternatively, the PostHTTP/ListenHTTP processors to pass 
the data and configuring the PostHTTP to include attributes.

If you are not sending the data directly to another NiFi but have some sort of intermediary step, you can
package the FlowFiles with their contents by using MergeContent. If you need each FlowFile packaged
separately, you can do that by setting both the min and max entries to 1. Otherwise, you can package multiple
together into a single file. For the Merge Strategy, use the Bin-Packing Algorithm and for the Merge Format,
choose the "FlowFile Stream, v3".

Sorry, these are some fairly long-winded answers, but I wanted to ensure that I didn't leave you with more
questions than answers :) If anything isn't clear, please let me know!

Thanks
-Mark

[1] https://issues.apache.org/jira/browse/NIFI-190
[2] https://issues.apache.org/jira/browse/NIFI-832



----------------------------------------
> Date: Sun, 9 Aug 2015 22:30:49 -0400
> Subject: NiFi Flow Usage questions
> From: edcrosbys@gmail.com
> To: dev@nifi.apache.org
>
> Do you have an example workflow of a REST API or external program that
> reads the flowfile contents and update attributes with the output?
>
> Can you have processors for child flows impact original flow?
> -Compressed file is input into system
> -Unpack file
> -Run all children files through external program
> -Make decision on original based on children attributes
>
> Last and not least-
> Is there a way to export and import attributes between different Nifi
> instances? You could have uuid.attributes and uuid.file to match them on
> the distance end.
>
> Thoughts?
>
> Thank you,
> Eric