You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Koji Kawamura <ij...@gmail.com> on 2018/11/02 02:27:26 UTC
Re: Status of MetricsReportingTask?
Hi Jon,
About reporting counter values, there is an existing JIRA for the
improvement idea to expose counters to Reporting task context. That
requires NiFi framework level improvements. I'd suggest taking a look
on it, and resuming discussion there if needed.
https://issues.apache.org/jira/browse/NIFI-3293
Although MetricsReportingTask currently only supports Graphite, the
component is well designed for generic reporting use-cases. I don't
think it is a legacy component. The underlying dropwizard metrics
project seems active, too.
https://github.com/dropwizard/metrics
I'm interested in what service implementation you're going to write.
Is it going to use one of already available reporters?
https://metrics.dropwizard.io/3.1.0/manual/third-party/
Thanks,
Koji
On Thu, Nov 1, 2018 at 3:17 AM Jon Logan <jm...@buffalo.edu> wrote:
>
> Hi All,
>
> I was looking at utilizing the MetricsReportingTask service, but I was
> wondering what the status of it is -- it seems to be lacking some features
> that I thought it'd have (ex. reporting counters), and I'm not sure there's
> an ability to extend the metrics being produced. Is this something that is
> still being worked on, or is a legacy component? We are going to have to
> write our own Service implementation, as the only supported one seems to be
> Graphite, and wanted to make sure we're not going down a legacy path.
>
> Thanks!
Re: Status of MetricsReportingTask?
Posted by Omer Hadari <ha...@gmail.com>.
Hi Jon. Just wanted to say that I have developed this ReportingTask since
we had a similar issue to yours, only we had to report metrics to graphite.
We are currently using it in production.
I tried to implement as generic design as possible, so reporting to other
endpoints (e.g. cloudwatch) should be easy enough.
I performed a quick google search about dropwizard-metrics cloudwatch
reporter implementation. Found many small projects, such as this one:
https://github.com/azagniotov/codahale-aggregated-metrics-cloudwatch-reporter.
It might fit your needs or you can at least take a look at it.
When I designed this I only had system and process group metrics in mind. I
was trying to be generic with the endpoints, so I used dropwizard-metrics
but I admit I haven't thought about allowing custom tasks to report custom
metrics. It shouldn't be too hard to implement even now, so let me know if
you need this option and I will try getting to submitting a contribution.
Feel free to let me know if you have any questions regarding this reporting
task.
On Fri, Nov 2, 2018 at 9:35 PM Bryan Bende <bb...@gmail.com> wrote:
> You should be able to make your own NAR with two modules like:
>
> nifi-cloudwatch-metrics - contains your service implementation with a
> provided dependency on nifi-metrics-reporter-service-api
> nifi-cloudwatch-metrics-nar - packages your implementation and has a
> NAR dependency on nifi-metrics-reporter-service-api-nar
>
> Similar to this example:
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices
>
> On Fri, Nov 2, 2018 at 3:24 PM Jon Logan <jm...@buffalo.edu> wrote:
> >
> > We were planning on having a reporter to pipe the metrics off to AWS
> > CloudWatch. The one issue being ran into -- and we've ran into it for
> other
> > components -- is the classloading. What is the recommended way of adding
> > alternate implementations of services (in this case
> MetricReporterService)?
> > We run into issues that I think stem from classloading isolation between
> > NARs, such that our custom implementations of services are not being
> > detected from the other NAR's context. We end up having to duplicate the
> > service into our custom NAR, but that hardly seems like a good idea.
> >
> > Thanks!
> > Jon
> >
> > On Thu, Nov 1, 2018 at 10:27 PM Koji Kawamura <ij...@gmail.com>
> > wrote:
> >
> > > Hi Jon,
> > >
> > > About reporting counter values, there is an existing JIRA for the
> > > improvement idea to expose counters to Reporting task context. That
> > > requires NiFi framework level improvements. I'd suggest taking a look
> > > on it, and resuming discussion there if needed.
> > > https://issues.apache.org/jira/browse/NIFI-3293
> > >
> > > Although MetricsReportingTask currently only supports Graphite, the
> > > component is well designed for generic reporting use-cases. I don't
> > > think it is a legacy component. The underlying dropwizard metrics
> > > project seems active, too.
> > > https://github.com/dropwizard/metrics
> > >
> > > I'm interested in what service implementation you're going to write.
> > > Is it going to use one of already available reporters?
> > > https://metrics.dropwizard.io/3.1.0/manual/third-party/
> > >
> > > Thanks,
> > > Koji
> > > On Thu, Nov 1, 2018 at 3:17 AM Jon Logan <jm...@buffalo.edu> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > I was looking at utilizing the MetricsReportingTask service, but I
> was
> > > > wondering what the status of it is -- it seems to be lacking some
> > > features
> > > > that I thought it'd have (ex. reporting counters), and I'm not sure
> > > there's
> > > > an ability to extend the metrics being produced. Is this something
> that
> > > is
> > > > still being worked on, or is a legacy component? We are going to
> have to
> > > > write our own Service implementation, as the only supported one
> seems to
> > > be
> > > > Graphite, and wanted to make sure we're not going down a legacy path.
> > > >
> > > > Thanks!
> > >
>
Re: Status of MetricsReportingTask?
Posted by Bryan Bende <bb...@gmail.com>.
You should be able to make your own NAR with two modules like:
nifi-cloudwatch-metrics - contains your service implementation with a
provided dependency on nifi-metrics-reporter-service-api
nifi-cloudwatch-metrics-nar - packages your implementation and has a
NAR dependency on nifi-metrics-reporter-service-api-nar
Similar to this example:
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices
On Fri, Nov 2, 2018 at 3:24 PM Jon Logan <jm...@buffalo.edu> wrote:
>
> We were planning on having a reporter to pipe the metrics off to AWS
> CloudWatch. The one issue being ran into -- and we've ran into it for other
> components -- is the classloading. What is the recommended way of adding
> alternate implementations of services (in this case MetricReporterService)?
> We run into issues that I think stem from classloading isolation between
> NARs, such that our custom implementations of services are not being
> detected from the other NAR's context. We end up having to duplicate the
> service into our custom NAR, but that hardly seems like a good idea.
>
> Thanks!
> Jon
>
> On Thu, Nov 1, 2018 at 10:27 PM Koji Kawamura <ij...@gmail.com>
> wrote:
>
> > Hi Jon,
> >
> > About reporting counter values, there is an existing JIRA for the
> > improvement idea to expose counters to Reporting task context. That
> > requires NiFi framework level improvements. I'd suggest taking a look
> > on it, and resuming discussion there if needed.
> > https://issues.apache.org/jira/browse/NIFI-3293
> >
> > Although MetricsReportingTask currently only supports Graphite, the
> > component is well designed for generic reporting use-cases. I don't
> > think it is a legacy component. The underlying dropwizard metrics
> > project seems active, too.
> > https://github.com/dropwizard/metrics
> >
> > I'm interested in what service implementation you're going to write.
> > Is it going to use one of already available reporters?
> > https://metrics.dropwizard.io/3.1.0/manual/third-party/
> >
> > Thanks,
> > Koji
> > On Thu, Nov 1, 2018 at 3:17 AM Jon Logan <jm...@buffalo.edu> wrote:
> > >
> > > Hi All,
> > >
> > > I was looking at utilizing the MetricsReportingTask service, but I was
> > > wondering what the status of it is -- it seems to be lacking some
> > features
> > > that I thought it'd have (ex. reporting counters), and I'm not sure
> > there's
> > > an ability to extend the metrics being produced. Is this something that
> > is
> > > still being worked on, or is a legacy component? We are going to have to
> > > write our own Service implementation, as the only supported one seems to
> > be
> > > Graphite, and wanted to make sure we're not going down a legacy path.
> > >
> > > Thanks!
> >
Re: Status of MetricsReportingTask?
Posted by Jon Logan <jm...@buffalo.edu>.
We were planning on having a reporter to pipe the metrics off to AWS
CloudWatch. The one issue being ran into -- and we've ran into it for other
components -- is the classloading. What is the recommended way of adding
alternate implementations of services (in this case MetricReporterService)?
We run into issues that I think stem from classloading isolation between
NARs, such that our custom implementations of services are not being
detected from the other NAR's context. We end up having to duplicate the
service into our custom NAR, but that hardly seems like a good idea.
Thanks!
Jon
On Thu, Nov 1, 2018 at 10:27 PM Koji Kawamura <ij...@gmail.com>
wrote:
> Hi Jon,
>
> About reporting counter values, there is an existing JIRA for the
> improvement idea to expose counters to Reporting task context. That
> requires NiFi framework level improvements. I'd suggest taking a look
> on it, and resuming discussion there if needed.
> https://issues.apache.org/jira/browse/NIFI-3293
>
> Although MetricsReportingTask currently only supports Graphite, the
> component is well designed for generic reporting use-cases. I don't
> think it is a legacy component. The underlying dropwizard metrics
> project seems active, too.
> https://github.com/dropwizard/metrics
>
> I'm interested in what service implementation you're going to write.
> Is it going to use one of already available reporters?
> https://metrics.dropwizard.io/3.1.0/manual/third-party/
>
> Thanks,
> Koji
> On Thu, Nov 1, 2018 at 3:17 AM Jon Logan <jm...@buffalo.edu> wrote:
> >
> > Hi All,
> >
> > I was looking at utilizing the MetricsReportingTask service, but I was
> > wondering what the status of it is -- it seems to be lacking some
> features
> > that I thought it'd have (ex. reporting counters), and I'm not sure
> there's
> > an ability to extend the metrics being produced. Is this something that
> is
> > still being worked on, or is a legacy component? We are going to have to
> > write our own Service implementation, as the only supported one seems to
> be
> > Graphite, and wanted to make sure we're not going down a legacy path.
> >
> > Thanks!
>