New NiFi Metric

["Owens, Mark" <jm...@evoforge.org>]

Hi,

I'm looking to implement a new metric in NiFi based upon some research performed by an intern last summer. It would be a 'time-to-capacity' estimate that would predict capacity overloads in dataflows. The goal would be to predict estimated time to overload within data flows and provide means to alert interested parties prior to that overload failure. The initial method would involve sampling real-time information directly from NiFi and calculating the rate of input/output for data connections using a sliding window in time. A value would be calculated estimating the remaining time until capacity overload by assuming a constant input/output difference over the sampled timeframe. This value would be updated at regular intervals using the latest input/output information. This method could be refined and improved as needed in time.

 I'm seeking comments on the level of difficulty you think this would entail, i.e., does it sound feasible? I'm also seeking a sense on how amenible the community would be to adding a new metric to NiFi and having it added to the stats page. Suggestions as to good portions of code where existing metric calculations are made and presented would be helpful as well.

Thanks,
Mark

RE: New NiFi Metric

["Owens, Mark" <jm...@evoforge.org>]

Thanks Jon,

I had not given thoughts to permissions yet as I'm at the very beginning stages of thinking about this. Good to be reminded of that aspect of the problem.

Mark

-----Original Message-----
From: Jon Logan <jm...@buffalo.edu> 
Sent: Friday, March 15, 2019 12:36 PM
To: dev@nifi.apache.org
Subject: Re: New NiFi Metric

It sounds interesting. The one potential issue I could see would be related to permissions, particularly if it's an implicit global thing.

Would this integrate with the existing Metrics capabilities (ex.
MetricsReportingTask)? Maybe there could be a way to decouple it into a metrics reporting task and have a UI Metrics Reporter Service that would allow for a more general metrics status page? This would allow you to take the same metric and publish it externally if desired (ex. to Cloudwatch), and trigger notification alarms, etc. Just a thought...

On Fri, Mar 15, 2019 at 8:46 AM Joe Witt <jo...@gmail.com> wrote:

> Mark
>
> Certainly sounds interesting and adding such a metric makes good sense.
>
> On Fri, Mar 15, 2019 at 11:39 AM Owens, Mark <jm...@evoforge.org> wrote:
>
> > Hi,
> >
> > I'm looking to implement a new metric in NiFi based upon some 
> > research performed by an intern last summer. It would be a 'time-to-capacity'
> > estimate that would predict capacity overloads in dataflows. The 
> > goal
> would
> > be to predict estimated time to overload within data flows and 
> > provide means to alert interested parties prior to that overload 
> > failure. The initial method would involve sampling real-time 
> > information directly from NiFi and calculating the rate of 
> > input/output for data connections using
> a
> > sliding window in time. A value would be calculated estimating the 
> > remaining time until capacity overload by assuming a constant
> input/output
> > difference over the sampled timeframe. This value would be updated 
> > at regular intervals using the latest input/output information. This 
> > method could be refined and improved as needed in time.
> >
> >  I'm seeking comments on the level of difficulty you think this 
> > would entail, i.e., does it sound feasible? I'm also seeking a sense 
> > on how amenible the community would be to adding a new metric to 
> > NiFi and having it added to the stats page. Suggestions as to good 
> > portions of code where existing metric calculations are made and 
> > presented would be helpful as well.
> >
> > Thanks,
> > Mark
> >
> >
>

Re: New NiFi Metric

[Jon Logan <jm...@buffalo.edu>]

It sounds interesting. The one potential issue I could see would be related
to permissions, particularly if it's an implicit global thing.

Would this integrate with the existing Metrics capabilities (ex.
MetricsReportingTask)? Maybe there could be a way to decouple it into a
metrics reporting task and have a UI Metrics Reporter Service that would
allow for a more general metrics status page? This would allow you to take
the same metric and publish it externally if desired (ex. to Cloudwatch),
and trigger notification alarms, etc. Just a thought...

On Fri, Mar 15, 2019 at 8:46 AM Joe Witt <jo...@gmail.com> wrote:

> Mark
>
> Certainly sounds interesting and adding such a metric makes good sense.
>
> On Fri, Mar 15, 2019 at 11:39 AM Owens, Mark <jm...@evoforge.org> wrote:
>
> > Hi,
> >
> > I'm looking to implement a new metric in NiFi based upon some research
> > performed by an intern last summer. It would be a 'time-to-capacity'
> > estimate that would predict capacity overloads in dataflows. The goal
> would
> > be to predict estimated time to overload within data flows and provide
> > means to alert interested parties prior to that overload failure. The
> > initial method would involve sampling real-time information directly from
> > NiFi and calculating the rate of input/output for data connections using
> a
> > sliding window in time. A value would be calculated estimating the
> > remaining time until capacity overload by assuming a constant
> input/output
> > difference over the sampled timeframe. This value would be updated at
> > regular intervals using the latest input/output information. This method
> > could be refined and improved as needed in time.
> >
> >  I'm seeking comments on the level of difficulty you think this would
> > entail, i.e., does it sound feasible? I'm also seeking a sense on how
> > amenible the community would be to adding a new metric to NiFi and having
> > it added to the stats page. Suggestions as to good portions of code where
> > existing metric calculations are made and presented would be helpful as
> > well.
> >
> > Thanks,
> > Mark
> >
> >
>

Re: New NiFi Metric

[Joe Witt <jo...@gmail.com>]

Mark

Certainly sounds interesting and adding such a metric makes good sense.

On Fri, Mar 15, 2019 at 11:39 AM Owens, Mark <jm...@evoforge.org> wrote:

> Hi,
>
> I'm looking to implement a new metric in NiFi based upon some research
> performed by an intern last summer. It would be a 'time-to-capacity'
> estimate that would predict capacity overloads in dataflows. The goal would
> be to predict estimated time to overload within data flows and provide
> means to alert interested parties prior to that overload failure. The
> initial method would involve sampling real-time information directly from
> NiFi and calculating the rate of input/output for data connections using a
> sliding window in time. A value would be calculated estimating the
> remaining time until capacity overload by assuming a constant input/output
> difference over the sampled timeframe. This value would be updated at
> regular intervals using the latest input/output information. This method
> could be refined and improved as needed in time.
>
>  I'm seeking comments on the level of difficulty you think this would
> entail, i.e., does it sound feasible? I'm also seeking a sense on how
> amenible the community would be to adding a new metric to NiFi and having
> it added to the stats page. Suggestions as to good portions of code where
> existing metric calculations are made and presented would be helpful as
> well.
>
> Thanks,
> Mark
>
>