You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Marroquin, Stuardo X. -ND" <St...@disney.com> on 2020/02/07 18:44:48 UTC
Monitoring/Alerting
Good afternoon,
I was wondering if there was a way to monitor the data flow into NiFi and also from NiFi to our storage device.
We have approximately 15 streams of files coming into NiFi at a rate of a file per minute, 24/7, 365 days, varying in size from 4MB to 12MB each. We need a way to ensure that we are notified if one of those streams stops.
I saw that you have PutEmail but there is not a whole lot of documentation on setting it up and I don't even know if that will meet our needs.
Any help you can provide on this would be greatly appreciated.
Kind Regards,
Stu Marroquin
Stu Marroquin
Information Management Senior Specialist
The Walt Disney Account
Atos North America
407-619-2498
stu.marroquin@atos.net<ma...@atos.net>
(DISC) 5601 Center Drive
Orlando, FL 32830
www.na.atos.net<http://www.na.atos.net/>
[Atos]
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted
Re: Monitoring/Alerting
Posted by Andy LoPresto <al...@apache.org>.
Have you tried using the MonitorActivity [1] processor? "Monitors the flow for activity and sends out an indicator when the flow has not had any data for some specified amount of time and again when the flow's activity is restored."
[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.MonitorActivity/ <https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.MonitorActivity/>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 7, 2020, at 11:26 AM, Emanuel Oliveira <em...@gmail.com> wrote:
>
> Hi,
>
> Simple suggestion to detect lag would be (imagining 1 single flow handling
> the 15 inbound streams):
>
> *Processor UpdateAttribute*
> Store State= store state locally
>
> add one attribute per stream to keep last ts
> last_stream_1_ts if stream=1 then ${now:toNumber()} else ${getStateValue("
> last_stream_1_ts")}
>
>
> processor RouteOnAttribute
> stream1_lag if ${now:toNumber()} - ${last_stream_1_ts} > 86400000 then
> 'yes' else 'no'.
>
>
> Sorry i used pseudo code -im in a restaurant :) - but just wanted to share
> how you generate Boolean attributes that one can then use to do whatever
> action needed.
>
> Hope this helps,
> Emanuel O.
>
>
>
>
>
>
>
>
>
>
>
> On Fri 7 Feb 2020, 18:47 Marroquin, Stuardo X. -ND, <
> Stuardo.X.Marroquin.-ND@disney.com <ma...@disney.com>> wrote:
>
>> Good afternoon,
>>
>>
>>
>> I was wondering if there was a way to monitor the data flow into NiFi and
>> also from NiFi to our storage device.
>>
>> We have approximately 15 streams of files coming into NiFi at a rate of a
>> file per minute, 24/7, 365 days, varying in size from 4MB to 12MB each. We
>> need a way to ensure that we are notified if one of those streams stops.
>>
>> I saw that you have PutEmail but there is not a whole lot of documentation
>> on setting it up and I don’t even know if that will meet our needs.
>>
>> Any help you can provide on this would be greatly appreciated.
>>
>>
>>
>> Kind Regards,
>>
>> Stu Marroquin
>>
>> [image: blue_strip]
>>
>> *Stu Marroquin*
>>
>> Information Management Senior Specialist
>>
>> The Walt Disney Account
>>
>> Atos North America
>>
>> 407-619-2498
>>
>> stu.marroquin@atos.net
>>
>> (DISC) 5601 Center Drive
>> Orlando, FL 32830
>>
>> www.na.atos.net
>>
>>
>>
>> [image: Atos]
>>
>>
>>
>> This e-mail and the documents attached are confidential and intended
>> solely for the addressee; it may also be privileged. If you receive this
>> e-mail in error, please notify the sender immediately and destroy it. As
>> its integrity cannot be secured on the Internet, the Atos group liability
>> cannot be triggered for the message content. Although the sender endeavors
>> to maintain a computer virus-free network, the sender does not warrant that
>> this transmission is virus-free and will not be liable for any damages
>> resulting from any virus transmitted
Re: Monitoring/Alerting
Posted by Emanuel Oliveira <em...@gmail.com>.
Hi,
Simple suggestion to detect lag would be (imagining 1 single flow handling
the 15 inbound streams):
*Processor UpdateAttribute*
Store State= store state locally
add one attribute per stream to keep last ts
last_stream_1_ts if stream=1 then ${now:toNumber()} else ${getStateValue("
last_stream_1_ts")}
processor RouteOnAttribute
stream1_lag if ${now:toNumber()} - ${last_stream_1_ts} > 86400000 then
'yes' else 'no'.
Sorry i used pseudo code -im in a restaurant :) - but just wanted to share
how you generate Boolean attributes that one can then use to do whatever
action needed.
Hope this helps,
Emanuel O.
On Fri 7 Feb 2020, 18:47 Marroquin, Stuardo X. -ND, <
Stuardo.X.Marroquin.-ND@disney.com> wrote:
> Good afternoon,
>
>
>
> I was wondering if there was a way to monitor the data flow into NiFi and
> also from NiFi to our storage device.
>
> We have approximately 15 streams of files coming into NiFi at a rate of a
> file per minute, 24/7, 365 days, varying in size from 4MB to 12MB each. We
> need a way to ensure that we are notified if one of those streams stops.
>
> I saw that you have PutEmail but there is not a whole lot of documentation
> on setting it up and I don’t even know if that will meet our needs.
>
> Any help you can provide on this would be greatly appreciated.
>
>
>
> Kind Regards,
>
> Stu Marroquin
>
> [image: blue_strip]
>
> *Stu Marroquin*
>
> Information Management Senior Specialist
>
> The Walt Disney Account
>
> Atos North America
>
> 407-619-2498
>
> stu.marroquin@atos.net
>
> (DISC) 5601 Center Drive
> Orlando, FL 32830
>
> www.na.atos.net
>
>
>
> [image: Atos]
>
>
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it. As
> its integrity cannot be secured on the Internet, the Atos group liability
> cannot be triggered for the message content. Although the sender endeavors
> to maintain a computer virus-free network, the sender does not warrant that
> this transmission is virus-free and will not be liable for any damages
> resulting from any virus transmitted
>
>
>