Re: Monitoring and Alerting

[Mike Heffner <mi...@solarwinds.cloud>]

Dropping in late on this thread, but we just released our Kafka metrics
reporter for AppOptics:
https://github.com/appoptics/kafka-appoptics-reporter

It ties into the Yammer Metrics reporting interface to capture metrics and
publish to Appoptics. Just build and drop the Jar into your Kafka libs
directory.


Cheers,

Mike

On Thu, Apr 26, 2018 at 12:35 AM, Anand, Uttam <Ut...@bnsf.com> wrote:

> You can use Prometheus for monitoring , AlertManager for alerting and
> Grafana to view the real time graphs by using the metrics scraped by
> Prometheus.
>
>
> For gathering the metrics of Kafka and Zookeeper , you can use jmx
> exporter. Also, all metrics will not be useful for humans to read and
> understand and it will be overhead for monitoring apps to process them. So
> you can whitelist the metrics you actually want.
>
>
> https://www.robustperception.io/monitoring-kafka-with-prometheus
>
>
> You can add jmx exporter as environment variable in container also.
> <https://www.robustperception.io/monitoring-kafka-with-prometheus>
>
> Monitoring Kafka with Prometheus | Robust Perception<https://www.
> robustperception.io/monitoring-kafka-with-prometheus/>
> www.robustperception.io
> We’ve previously looked at how to monitor Cassandra with Prometheus. Let’s
> see the process for getting metrics from another popular Java application,
> Kafka. Similar to what we did for Cassandra, we download Kafka, the JMX
> exporter and the config file:
>
>
>
>
> ________________________________
> From: Will Weber <rw...@gmail.com>
> Sent: Wednesday, April 25, 2018 11:27:40 PM
> To: users@kafka.apache.org
> Subject: Re: Monitoring and Alerting
>
> <center><p style="color:red; font-size:24px;"><b>EXTERNAL
> EMAIL</b></p></center>
> "Thirding" the use of prometheus, recommend reading this blog post as well:
> https://www.robustperception.io/monitoring-kafka-with-prometheus
>
> On Wed, Apr 25, 2018 at 11:39 PM, Jonathan Bethune <
> jonathan.bethune@instaclustr.com> wrote:
>
> > I also recommend Prometheus. Works great with JMX for Kafka or any Java
> > service.
> >
> > Datadog is also fine if you want simple and don't spending money. Easy to
> > integrate with alerting systems and great visualization.
> >
> > They have a good blogpost about integrating with Kafka and ZooKeeper:
> > https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/
> >
> > On 26 April 2018 at 08:02, Stanislav Antic <st...@gmail.com>
> > wrote:
> >
> > > I recommend setting up Prometheus which have Zookeeper exporter (helper
> > > program which gives metrics from outside software) and you can use JMX
> > > exporter with Kafka.
> > >
> > > You can find example configs in their repo, which is pretty good and
> you
> > > also have already done Grafana dashboards (https://grafana.com/
> > dashboards
> > > ):
> > > https://github.com/prometheus/jmx_exporter/tree/master/example_configs
> > >
> > >
> > > On Thu, Apr 26, 2018 at 12:05 AM, Arunkumar <pm_arunkumar@yahoo.com.
> > > invalid>
> > > wrote:
> > >
> > > > HI All
> > > > I am working on setting up Monitoring and alerting for our production
> > > > cluster. As of now we have a cluster of 3 zookeeper and 3 kafka
> Brokers
> > > > which will expand later.
> > > > We are planning for basic metrics (important ones) on which we need
> to
> > > > alert. We are in a process of developing alerting system for our
> > > cluster. I
> > > > googled and did not get much detail as needed.
> > > > I see many say the following ones are important and I understand
> these
> > > are
> > > > aggregate metrics across cluster (I may not be
> > right).kafka.server:type=
> > > > ReplicaManager,name=UnderReplicatedPartitionskafka.controller:type=
> > > > KafkaController,name=OfflinePartitionsCountkafka.controller:type=
> > > > KafkaController,name=ActiveControllerCount
> > > > But I am looking for broker,topic etc metrics to alert. Can any one
> > > > provide info or point to any docs on this regards will be of great
> > > > help.            Thanks in advance
> > > >
> > > > ThanksArunkumar Pichaimuthu, PMP
> > >
> > >
> > >
> > >
> > > --
> > > Stanislav Antic
> > >
> >
> >
> >
> > --
> >
> > *Jonathan Bethune - **Senior Consultant*
> >
> > JP: +81 70 4069 4357
> >
> > <https://www.facebook.com/instaclustr>   <https://twitter.com/
> instaclustr>
> > <https://www.linkedin.com/company/instaclustr>
> >
> > Read our latest technical blog posts here
> > <https://www.instaclustr.com/blog/>. This email has been sent on behalf
> > of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
> > email and any attachments may contain confidential and legally
> > privileged information.  If you are not the intended recipient, do not
> copy
> > or disclose its content, but please reply to this email immediately and
> > highlight the error to the sender and then immediately delete the
> message.
> >
>