You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by "Dobbernack, Harald (Key-Work)" <ha...@key-work.de> on 2020/03/04 13:10:53 UTC

Logging/Monitoring of Invalid Processors at NiFi Startup

Our standalone Nifi 1.11.1 on Debian 10.2 will on service startup not throw an Error or Warning into the NiFi log if it deams a processor invalid, for example if a samba mount is not available and the listfile or getfile processors cannot reach the mount. If, on the other hand, the processors are running and the connection to the mount gets lost then we will see Error entrances in the NiFi App log, which we then can put to use to alert us.

We had thought to let Nifi report Errors to us via push mail via taillog processor on it's own log, but in case of unreachable mount at service startup it wouldn't be able to alert us that something is wrong.

Is it possible to log invalid processors at service startup? Or how Do you monitor or report failures of this type?

Thank you,
Harald


Harald Dobbernack
Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | https://www.key-work.de | Datenschutz<https://www.key-work.de/de/footer/datenschutz.html>
Fon: +49-721-78203-264 | E-Mail: harald.dobbernack@key-work.de | Fax: +49-721-78203-10

Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
Gesch?ftsf?hrer: Andreas Stappert, Tobin Wotring

AW: Logging/Monitoring of Invalid Processors at NiFi Startup

Posted by "Dobbernack, Harald (Key-Work)" <ha...@key-work.de>.

Hi Pierre,

thank you for these ideas! I'm looking forward to trying that out.

Merci beaucoup!
Harald

Von: Pierre Villard <pi...@gmail.com>
Gesendet: Mittwoch, 4. März 2020 19:46
An: users@nifi.apache.org
Betreff: Re: Logging/Monitoring of Invalid Processors at NiFi Startup

Hi,

You could use the SiteToSiteStatusReportingTask and leverage the "runStatus" field to list all the components (processors, controller services, etc) that are invalid. You could even use the QueryNiFiReportingTask to directly filter only the invalid processors and sink this information somewhere.

Hope this helps,
Pierre

Le mer. 4 mars 2020 à 16:44, <ma...@swisscom.com> a écrit :
With logback.xml you can finetune logmessages, but don't ask me the details __.

Cheers Josef

On 04.03.20, 16:35, "Dobbernack, Harald (Key-Work)" <ma...@key-work.de> wrote:

Hi Josef,

thank you for your input! We planned on merging/aggregating the errors into one mail an hour (so the length of the mail would only be caused by the distinct number of error types in the timeframe), but of course I'll check with my splunk colleagues! We still have the problem though that the invalid processors are not logged into the nifi-app.log - or is there a way to enable this?

Thank you,
Harald

Von: mailto:Josef.Zahner1@swisscom.com <ma...@swisscom.com>
Gesendet: Mittwoch, 4. März 2020 16:19
An: mailto:users@nifi.apache.org
Betreff: Re: Logging/Monitoring of Invalid Processors at NiFi Startup

Hi Harald,

I can tell you only what we do, we are sending the whole nifi-app.log to splunk and scan there specific for alarms/warnings. We don’t use e-mail notification as it doesn’t help too much. In the nifi-app.log you would see as well startup issues, so we just focus on that instead.

In my eyes e-mail isn’t the right medium to alert/monitor, eg. If you have massiv issues it would flood your e-mail account completely with warnings – and I don’t think that you want that.

Cheers Josef

From: "Dobbernack, Harald (Key-Work)" <ma...@key-work.de>
Reply to: "mailto:mailto:users@nifi.apache.org" <ma...@nifi.apache.org>
Date: Wednesday, 4 March 2020 at 14:11
To: "mailto:mailto:users@nifi.apache.org" <ma...@nifi.apache.org>
Subject: Logging/Monitoring of Invalid Processors at NiFi Startup

Our standalone Nifi 1.11.1 on Debian 10.2 will on service startup not throw an Error or Warning into the NiFi log if it deams a processor invalid, for example if a samba mount is not available and the listfile or getfile processors cannot reach the mount. If, on the other hand, the processors are running and the connection to the mount gets lost then we will see Error entrances in the NiFi App log, which we then can put to use to alert us.

We had thought to let Nifi report Errors to us via push mail via taillog processor on it’s own log, but in case of unreachable mount at service startup it wouldn’t be able to alert us that something is wrong.

Is it possible to log invalid processors at service startup? Or how Do you monitor or report failures of this type?

Thank you,
Harald
--

Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
Geschäftsführer: Andreas Stappert, Tobin Wotring

Re: Logging/Monitoring of Invalid Processors at NiFi Startup

Posted by Pierre Villard <pi...@gmail.com>.

Hi,

You could use the SiteToSiteStatusReportingTask and leverage the
"runStatus" field to list all the components (processors, controller
services, etc) that are invalid. You could even use
the QueryNiFiReportingTask to directly filter only the invalid processors
and sink this information somewhere.

Hope this helps,
Pierre

Le mer. 4 mars 2020 à 16:44, <Jo...@swisscom.com> a écrit :

> With logback.xml you can finetune logmessages, but don't ask me the
> details __.
>
> Cheers Josef
>
>
> On 04.03.20, 16:35, "Dobbernack, Harald (Key-Work)" <
> harald.dobbernack@key-work.de> wrote:
>
>     Hi Josef,
>
>     thank you for your input! We planned on merging/aggregating the errors
> into one mail an hour (so the length of the mail would only be caused by
> the distinct number of error types in the timeframe), but of course I'll
> check with my splunk colleagues!  We still have the problem though that the
> invalid processors are not logged into the nifi-app.log - or is there a way
> to enable this?
>
>     Thank you,
>     Harald
>
>
>
>     Von: Josef.Zahner1@swisscom.com <Jo...@swisscom.com>
>     Gesendet: Mittwoch, 4. März 2020 16:19
>     An: users@nifi.apache.org
>     Betreff: Re: Logging/Monitoring of Invalid Processors at NiFi Startup
>
>     Hi Harald,
>
>     I can tell you only what we do, we are sending the whole nifi-app.log
> to splunk and scan there specific for alarms/warnings. We don’t use e-mail
> notification as it doesn’t help too much. In the nifi-app.log you would see
> as well startup issues, so we just focus on that instead.
>
>     In my eyes e-mail isn’t the right medium to alert/monitor, eg. If you
> have massiv issues it would flood your e-mail account completely with
> warnings – and I don’t think that you want that.
>
>     Cheers Josef
>
>
>     From: "Dobbernack, Harald (Key-Work)" <mailto:
> harald.dobbernack@key-work.de>
>     Reply to: "mailto:users@nifi.apache.org" <mailto:users@nifi.apache.org
> >
>     Date: Wednesday, 4 March 2020 at 14:11
>     To: "mailto:users@nifi.apache.org" <ma...@nifi.apache.org>
>     Subject: Logging/Monitoring of Invalid Processors at NiFi Startup
>
>     Our standalone Nifi 1.11.1 on Debian 10.2 will on service startup not
> throw an Error or Warning into the NiFi log if it deams a processor
> invalid, for example if a samba mount is not available and the listfile or
> getfile processors cannot reach the mount. If, on the other hand, the
> processors are running and the connection to the mount gets lost then we
> will see Error entrances in the NiFi App log, which we then can put to use
> to alert us.
>
>     We had thought to let Nifi report Errors to us via push mail via
> taillog processor on it’s own log, but in case of unreachable mount at
> service startup it wouldn’t be able to alert us that something is wrong.
>
>     Is it possible to log invalid processors at service startup? Or how Do
> you monitor or report failures of this type?
>
>     Thank you,
>     Harald
>
>      --
>
>
>      Harald Dobbernack
>     Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe |
> Germany | https://www.key-work.de | Datenschutz<
> https://www.key-work.de/de/footer/datenschutz.html>
>     Fon: +49-721-78203-264 | E-Mail: harald.dobbernack@key-work.de | Fax:
> +49-721-78203-10
>
>     Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
>     Geschäftsführer: Andreas Stappert, Tobin Wotring
>
>
>

Re: Logging/Monitoring of Invalid Processors at NiFi Startup

Posted by Jo...@swisscom.com.

With logback.xml you can finetune logmessages, but don't ask me the details __.

Cheers Josef


On 04.03.20, 16:35, "Dobbernack, Harald (Key-Work)" <ha...@key-work.de> wrote:

    Hi Josef,
    
    thank you for your input! We planned on merging/aggregating the errors into one mail an hour (so the length of the mail would only be caused by the distinct number of error types in the timeframe), but of course I'll check with my splunk colleagues!  We still have the problem though that the invalid processors are not logged into the nifi-app.log - or is there a way to enable this?
    
    Thank you,
    Harald
    
    
    
    Von: Josef.Zahner1@swisscom.com <Jo...@swisscom.com>
    Gesendet: Mittwoch, 4. März 2020 16:19
    An: users@nifi.apache.org
    Betreff: Re: Logging/Monitoring of Invalid Processors at NiFi Startup
    
    Hi Harald,
    
    I can tell you only what we do, we are sending the whole nifi-app.log to splunk and scan there specific for alarms/warnings. We don’t use e-mail notification as it doesn’t help too much. In the nifi-app.log you would see as well startup issues, so we just focus on that instead.
    
    In my eyes e-mail isn’t the right medium to alert/monitor, eg. If you have massiv issues it would flood your e-mail account completely with warnings – and I don’t think that you want that.
    
    Cheers Josef
    
    
    From: "Dobbernack, Harald (Key-Work)" <ma...@key-work.de>
    Reply to: "mailto:users@nifi.apache.org" <ma...@nifi.apache.org>
    Date: Wednesday, 4 March 2020 at 14:11
    To: "mailto:users@nifi.apache.org" <ma...@nifi.apache.org>
    Subject: Logging/Monitoring of Invalid Processors at NiFi Startup
    
    Our standalone Nifi 1.11.1 on Debian 10.2 will on service startup not throw an Error or Warning into the NiFi log if it deams a processor invalid, for example if a samba mount is not available and the listfile or getfile processors cannot reach the mount. If, on the other hand, the processors are running and the connection to the mount gets lost then we will see Error entrances in the NiFi App log, which we then can put to use to alert us.
    
    We had thought to let Nifi report Errors to us via push mail via taillog processor on it’s own log, but in case of unreachable mount at service startup it wouldn’t be able to alert us that something is wrong.
    
    Is it possible to log invalid processors at service startup? Or how Do you monitor or report failures of this type?
    
    Thank you,
    Harald
    
     --
    
    
     Harald Dobbernack
    Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | https://www.key-work.de | Datenschutz<https://www.key-work.de/de/footer/datenschutz.html>
    Fon: +49-721-78203-264 | E-Mail: harald.dobbernack@key-work.de | Fax: +49-721-78203-10
    
    Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
    Geschäftsführer: Andreas Stappert, Tobin Wotring

AW: Logging/Monitoring of Invalid Processors at NiFi Startup

Posted by "Dobbernack, Harald (Key-Work)" <ha...@key-work.de>.

Hi Josef,

thank you for your input! We planned on merging/aggregating the errors into one mail an hour (so the length of the mail would only be caused by the distinct number of error types in the timeframe), but of course I'll check with my splunk colleagues!  We still have the problem though that the invalid processors are not logged into the nifi-app.log - or is there a way to enable this?

Thank you,
Harald



Von: Josef.Zahner1@swisscom.com <Jo...@swisscom.com>
Gesendet: Mittwoch, 4. März 2020 16:19
An: users@nifi.apache.org
Betreff: Re: Logging/Monitoring of Invalid Processors at NiFi Startup

Hi Harald,

I can tell you only what we do, we are sending the whole nifi-app.log to splunk and scan there specific for alarms/warnings. We don’t use e-mail notification as it doesn’t help too much. In the nifi-app.log you would see as well startup issues, so we just focus on that instead.

In my eyes e-mail isn’t the right medium to alert/monitor, eg. If you have massiv issues it would flood your e-mail account completely with warnings – and I don’t think that you want that.

Cheers Josef


From: "Dobbernack, Harald (Key-Work)" <ma...@key-work.de>
Reply to: "mailto:users@nifi.apache.org" <ma...@nifi.apache.org>
Date: Wednesday, 4 March 2020 at 14:11
To: "mailto:users@nifi.apache.org" <ma...@nifi.apache.org>
Subject: Logging/Monitoring of Invalid Processors at NiFi Startup

Our standalone Nifi 1.11.1 on Debian 10.2 will on service startup not throw an Error or Warning into the NiFi log if it deams a processor invalid, for example if a samba mount is not available and the listfile or getfile processors cannot reach the mount. If, on the other hand, the processors are running and the connection to the mount gets lost then we will see Error entrances in the NiFi App log, which we then can put to use to alert us.

We had thought to let Nifi report Errors to us via push mail via taillog processor on it’s own log, but in case of unreachable mount at service startup it wouldn’t be able to alert us that something is wrong.

Is it possible to log invalid processors at service startup? Or how Do you monitor or report failures of this type?

Thank you,
Harald

 --


 Harald Dobbernack
Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | https://www.key-work.de | Datenschutz<https://www.key-work.de/de/footer/datenschutz.html>
Fon: +49-721-78203-264 | E-Mail: harald.dobbernack@key-work.de | Fax: +49-721-78203-10

Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
Geschäftsführer: Andreas Stappert, Tobin Wotring

Re: Logging/Monitoring of Invalid Processors at NiFi Startup

Posted by Jo...@swisscom.com.

Hi Harald,

I can tell you only what we do, we are sending the whole nifi-app.log to splunk and scan there specific for alarms/warnings. We don’t use e-mail notification as it doesn’t help too much. In the nifi-app.log you would see as well startup issues, so we just focus on that instead.

In my eyes e-mail isn’t the right medium to alert/monitor, eg. If you have massiv issues it would flood your e-mail account completely with warnings – and I don’t think that you want that.

Cheers Josef


From: "Dobbernack, Harald (Key-Work)" <ha...@key-work.de>
Reply to: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 4 March 2020 at 14:11
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Logging/Monitoring of Invalid Processors at NiFi Startup

Our standalone Nifi 1.11.1 on Debian 10.2 will on service startup not throw an Error or Warning into the NiFi log if it deams a processor invalid, for example if a samba mount is not available and the listfile or getfile processors cannot reach the mount. If, on the other hand, the processors are running and the connection to the mount gets lost then we will see Error entrances in the NiFi App log, which we then can put to use to alert us.

We had thought to let Nifi report Errors to us via push mail via taillog processor on it’s own log, but in case of unreachable mount at service startup it wouldn’t be able to alert us that something is wrong.

Is it possible to log invalid processors at service startup? Or how Do you monitor or report failures of this type?

Thank you,
Harald


Harald Dobbernack
Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | https://www.key-work.de | Datenschutz<https://www.key-work.de/de/footer/datenschutz.html>
Fon: +49-721-78203-264 | E-Mail: harald.dobbernack@key-work.de | Fax: +49-721-78203-10

Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
Geschäftsführer: Andreas Stappert, Tobin Wotring