You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by James Farrington <ni...@gmail.com> on 2017/08/07 19:09:18 UTC
FlowFile Logging
Hi everyone,
Whenever there is an error in a flow and some flow file is not processed
properly, I need to retrieve the Flow File UUID and the size of that file.
I checked the nifi-app.log file and it seemed that most (if not all) of the
time this information was being sent there on a processor error. Does
anyone know if this will always be the case? Or if there is a better way to
capture all flow files that failed to process?
Any help would be great!
Thank you.
Re: FlowFile Logging
Posted by James Farrington <ni...@gmail.com>.
ah that sounds perfect for what I need to do. Thank you!
On Tue, Aug 8, 2017 at 7:20 PM, Andy LoPresto <al...@apache.org> wrote:
> You can add an appender in the conf/logback.xml file which handles
> “org.apache.nifi” and anything of ERROR level and writes to a separate log
> file, which you can then monitor/parse/send wherever you like. This will
> filter out the normal operations logging from the failures you are
> interested in. Unfortunately, by default the full stacktraces and some
> other error information will be included here as well.
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Aug 8, 2017, at 1:09 PM, James Farrington <ni...@gmail.com> wrote:
>
> Hi Andy,
>
> I am trying to catch any error that happens from any processor in my flow.
> So adding a PutFile for each and every processor would not be ideal. And I
> don't need the data to be transformed into a usable type. I am passing this
> data to Logstash as a csv file (so raw data is just fine). Any thoughts?
>
> Thank you,
> James
>
> On Mon, Aug 7, 2017 at 11:35 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Hi James,
>
> The app log will definitely contain a lot of relevant information about
> flowfile failure, but you can also make this easier for yourself by routing
> the failure connection of the relevant processor to a PutFile/PutEmail
> processor which outputs the flowfile UUID and content claim size to a
> special error destination. This means you won’t have to parse the app log
> and filter the non-error information. You may also want to look at the
> SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to
> transform the metadata (errors = bulletins, provenance for failure events)
> into raw data that you can then operate on directly.
>
> Obviously, you can also have that failure connection go to other
> processors, back to the original to retry, etc.
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Aug 7, 2017, at 12:09 PM, James Farrington <ni...@gmail.com> wrote:
>
> Hi everyone,
>
> Whenever there is an error in a flow and some flow file is not processed
> properly, I need to retrieve the Flow File UUID and the size of that file.
> I checked the nifi-app.log file and it seemed that most (if not all) of the
> time this information was being sent there on a processor error. Does
> anyone know if this will always be the case? Or if there is a better way to
> capture all flow files that failed to process?
>
> Any help would be great!
>
> Thank you.
>
>
>
Re: FlowFile Logging
Posted by Andy LoPresto <al...@apache.org>.
You can add an appender in the conf/logback.xml file which handles “org.apache.nifi” and anything of ERROR level and writes to a separate log file, which you can then monitor/parse/send wherever you like. This will filter out the normal operations logging from the failures you are interested in. Unfortunately, by default the full stacktraces and some other error information will be included here as well.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 8, 2017, at 1:09 PM, James Farrington <ni...@gmail.com> wrote:
>
> Hi Andy,
>
> I am trying to catch any error that happens from any processor in my flow.
> So adding a PutFile for each and every processor would not be ideal. And I
> don't need the data to be transformed into a usable type. I am passing this
> data to Logstash as a csv file (so raw data is just fine). Any thoughts?
>
> Thank you,
> James
>
> On Mon, Aug 7, 2017 at 11:35 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>> wrote:
>
>> Hi James,
>>
>> The app log will definitely contain a lot of relevant information about
>> flowfile failure, but you can also make this easier for yourself by routing
>> the failure connection of the relevant processor to a PutFile/PutEmail
>> processor which outputs the flowfile UUID and content claim size to a
>> special error destination. This means you won’t have to parse the app log
>> and filter the non-error information. You may also want to look at the
>> SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to
>> transform the metadata (errors = bulletins, provenance for failure events)
>> into raw data that you can then operate on directly.
>>
>> Obviously, you can also have that failure connection go to other
>> processors, back to the original to retry, etc.
>>
>> Andy LoPresto
>> alopresto@apache.org
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Aug 7, 2017, at 12:09 PM, James Farrington <ni...@gmail.com> wrote:
>>
>> Hi everyone,
>>
>> Whenever there is an error in a flow and some flow file is not processed
>> properly, I need to retrieve the Flow File UUID and the size of that file.
>> I checked the nifi-app.log file and it seemed that most (if not all) of the
>> time this information was being sent there on a processor error. Does
>> anyone know if this will always be the case? Or if there is a better way to
>> capture all flow files that failed to process?
>>
>> Any help would be great!
>>
>> Thank you.
Re: FlowFile Logging
Posted by James Farrington <ni...@gmail.com>.
Hi Andy,
I am trying to catch any error that happens from any processor in my flow.
So adding a PutFile for each and every processor would not be ideal. And I
don't need the data to be transformed into a usable type. I am passing this
data to Logstash as a csv file (so raw data is just fine). Any thoughts?
Thank you,
James
On Mon, Aug 7, 2017 at 11:35 PM, Andy LoPresto <al...@apache.org> wrote:
> Hi James,
>
> The app log will definitely contain a lot of relevant information about
> flowfile failure, but you can also make this easier for yourself by routing
> the failure connection of the relevant processor to a PutFile/PutEmail
> processor which outputs the flowfile UUID and content claim size to a
> special error destination. This means you won’t have to parse the app log
> and filter the non-error information. You may also want to look at the
> SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to
> transform the metadata (errors = bulletins, provenance for failure events)
> into raw data that you can then operate on directly.
>
> Obviously, you can also have that failure connection go to other
> processors, back to the original to retry, etc.
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Aug 7, 2017, at 12:09 PM, James Farrington <ni...@gmail.com> wrote:
>
> Hi everyone,
>
> Whenever there is an error in a flow and some flow file is not processed
> properly, I need to retrieve the Flow File UUID and the size of that file.
> I checked the nifi-app.log file and it seemed that most (if not all) of the
> time this information was being sent there on a processor error. Does
> anyone know if this will always be the case? Or if there is a better way to
> capture all flow files that failed to process?
>
> Any help would be great!
>
> Thank you.
>
>
>
Re: FlowFile Logging
Posted by Andy LoPresto <al...@apache.org>.
Hi James,
The app log will definitely contain a lot of relevant information about flowfile failure, but you can also make this easier for yourself by routing the failure connection of the relevant processor to a PutFile/PutEmail processor which outputs the flowfile UUID and content claim size to a special error destination. This means you won’t have to parse the app log and filter the non-error information. You may also want to look at the SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to transform the metadata (errors = bulletins, provenance for failure events) into raw data that you can then operate on directly.
Obviously, you can also have that failure connection go to other processors, back to the original to retry, etc.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 7, 2017, at 12:09 PM, James Farrington <ni...@gmail.com> wrote:
>
> Hi everyone,
>
> Whenever there is an error in a flow and some flow file is not processed
> properly, I need to retrieve the Flow File UUID and the size of that file.
> I checked the nifi-app.log file and it seemed that most (if not all) of the
> time this information was being sent there on a processor error. Does
> anyone know if this will always be the case? Or if there is a better way to
> capture all flow files that failed to process?
>
> Any help would be great!
>
> Thank you.