Outputting flowfiles to disk

[Kiran <b....@gmail.com>]

Hello,

Within my NiFi flows for the error scenarios I would really like the 
option of outputting the flow file to an error directory (the outputted 
file contains the flow file contents and well as the attributes).

This way once the error has been resolved I can replay the FlowFile by 
reading it back in which would read the contents as well as the flow 
file attributes.

From looking through the processor list the only way I can see to do 
this is by:
1. Using the AttributesToJson to output the attributes and separately 
output the flowfile contents. Then read both the contents and JSON file 
back in and parse the JSON back into attributes.
2. Use a groovy script within the ExecuteScript processor to combine the 
attributes and contents together. Then output the results to disk. When 
reading it back in use another groovy script to parse the file and 
populate the attributes and contents.

My preferred option is number 2.

Can I confirm that I haven't missed anything obvious.

Thanks,

Brian



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Outputting flowfiles to disk

[Russell Bateman <ru...@windofkeltia.com>]

Or, 3?

    ---------------------> MergeContent
                                   +---------> PutFile
    AttributesToJson ----> MergeContent


Or, 4?

    Join the ranks of custom processor writers and write one to do
    exactly what you want--good idea if this a pretty permanent part of
    your roadmap.


Hope this helps.

Russ

On 02/15/2017 03:18 PM, Kiran wrote:
> Hello,
> Within my NiFi flows for the error scenarios I would really like the 
> option of outputting the flow file to an error directory (the 
> outputted file contains the flow file contents and well as the 
> attributes).
> This way once the error has been resolved I can replay the FlowFile by 
> reading it back in which would read the contents as well as the flow 
> file attributes.
> From looking through the processor list the only way I can see to do 
> this is by:
> 1. Using the AttributesToJson to output the attributes and separately 
> output the flowfile contents. Then read both the contents and JSON 
> file back in and parse the JSON back into attributes.
> 2. Use a groovy script within the ExecuteScript processor to combine 
> the attributes and contents together. Then output the results to disk. 
> When reading it back in use another groovy script to parse the file 
> and populate the attributes and contents.
> My preferred option is number 2.
> Can I confirm that I haven't missed anything obvious.
> Thanks,
> Brian
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=icon> 
> 	Virus-free. www.avast.com 
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=link> 
>
>

Re: Outputting flowfiles to disk

[Joe Witt <jo...@gmail.com>]

Hello Brian

A good way to do this pattern is to use 'MergeContent' and set the Merge
Format to Flow File Stream, v3.  This way the errors are bundled together
nicely/efficiently.  Ensure it gets a unique filename whenever it is dumped
on disk too.

When you read that bundle/file back off disk you put it through
UnpackContent with flowfile stream v3.

There are a lot of strategies for this sort of dead-letter queue behavior
so if you want to talk through it more feel free to do so.  But for what
you asked the above will do it.

Thanks
Joe

On Wed, Feb 15, 2017 at 5:18 PM, Kiran <b....@gmail.com>
wrote:

> Hello,
>
> Within my NiFi flows for the error scenarios I would really like the
> option of outputting the flow file to an error directory (the outputted
> file contains the flow file contents and well as the attributes).
>
> This way once the error has been resolved I can replay the FlowFile by
> reading it back in which would read the contents as well as the flow file
> attributes.
>
> From looking through the processor list the only way I can see to do this
> is by:
> 1. Using the AttributesToJson to output the attributes and separately
> output the flowfile contents. Then read both the contents and JSON file
> back in and parse the JSON back into attributes.
> 2. Use a groovy script within the ExecuteScript processor to combine the
> attributes and contents together. Then output the results to disk. When
> reading it back in use another groovy script to parse the file and populate
> the attributes and contents.
>
> My preferred option is number 2.
>
> Can I confirm that I haven't missed anything obvious.
>
> Thanks,
>
> Brian
>
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=icon> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=link>
>