You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Sullivan, Tim" <TI...@homedepot.com> on 2019/12/20 12:51:59 UTC
Re: [EXTERNAL] Re: Health Check
Wonderful, I assume that it is only in the latest release, correct? I’m running 1.1.0 so I’m thinking I’d have to upgrade to get the benefits?
Tim Sullivan
Be Well
Sr. Software Engineer
Supply Chain – IT | Data & Analytics
The Home Depot
2250 Newmarket Parkway | Atlanta, GA 30339
Work Cell 470-455-8346
Personal Cell 678-525-2583
Home Phone: 770-945-3315
[signature_522277300]
From: Liam Clarke <li...@adscale.co.nz>
Reply-To: "users@kafka.apache.org" <us...@kafka.apache.org>
Date: Thursday, December 19, 2019 at 6:13 PM
To: "users@kafka.apache.org" <us...@kafka.apache.org>
Subject: [EXTERNAL] Re: Health Check
We've wired the Prometheus JMX exporter java-agent into our Kafka
processes, we've then set up alerts in Prometheus for several key metrics
to monitor cluster health (under replicated partitions/offline partitions,
request handler avg idle percent to measure load etc.) - and because the
JMX exporter runs within the Kafka process, any sustained failure to scrape
also sets off an alert.
Cheers,
Liam Clarke
On Thu, Dec 19, 2019 at 12:42 AM Miguel Silvestre <ms...@gmail.com>>
wrote:
A simple tcp connection to kafka port (9092) should be enough no?
--
Miguel Silvestre
On Tue, Dec 17, 2019 at 8:33 PM M. Manna <ma...@gmail.com>> wrote:
> Jai,
>
> On Tue, 17 Dec 2019 at 17:33, Jai Nanda Kumar <ja...@idexcel.com>>
> wrote:
>
> > Hi,
> >
> > How to perform a health check on a running Kafka server in AWS EC2
> > server.
> >
>
> SHouldn't this be part of your liveness probe? Or, are you trying to do
> this adhoc (not how folks do it anyway)?
>
> >
> > Thanks and Regards,
> > A. Jai Nanda Kumar,
> > Senior Software Engineer,
> > 9066260799.
> > Confidentiality Notice: This e-mail message, including any attachments,
> is
> > for the sole use of the intended recipient(s) and may contain
> confidential
> > and privileged information. Any unauthorized review, use, disclosure or
> > distribution is prohibited. If you are not the intended recipient,
please
> > contact the sender by reply e-mail and destroy all copies of the
original
> > message.
> >
>
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: [EXTERNAL] Re: Health Check
Posted by "M. Manna" <ma...@gmail.com>.
This is really helpful. Thanks for sharing this with the community.
On Sat, 21 Dec 2019 at 19:29, Alex Woolford <al...@woolford.io> wrote:
> Not sure if this is helpful, Tim.
>
> I recently recorded a video that shows the gist of monitoring Kafka with
> Prometheus: https://www.youtube.com/watch?v=nk3sk1LO7Bo
>
> Cheers,
>
> Alex
>
> On Sat, Dec 21, 2019 at 12:13 PM Liam Clarke <li...@adscale.co.nz>
> wrote:
>
> > Nope, it's a third party project:
> > https://github.com/prometheus/jmx_exporter
> >
> > We're using it with an 0.11 cluster.
> >
> > On Sat, 21 Dec. 2019, 1:52 am Sullivan, Tim, <TIM_SULLIVAN@homedepot.com
> >
> > wrote:
> >
> > >
> > > Wonderful, I assume that it is only in the latest release,
> > > correct? I’m running 1.1.0 so I’m thinking I’d have to upgrade to get
> the
> > > benefits?
> > >
> > >
> > > Tim Sullivan
> > >
> > > Be Well
> > >
> > >
> > >
> > > Sr. Software Engineer
> > > Supply Chain – IT | Data & Analytics
> > > The Home Depot
> > > 2250 Newmarket Parkway | Atlanta, GA 30339
> > >
> > > Work Cell 470-455-8346
> > > Personal Cell 678-525-2583
> > > Home Phone: 770-945-3315
> > >
> > >
> > > *From: *Liam Clarke <li...@adscale.co.nz>
> > > *Reply-To: *"users@kafka.apache.org" <us...@kafka.apache.org>
> > > *Date: *Thursday, December 19, 2019 at 6:13 PM
> > > *To: *"users@kafka.apache.org" <us...@kafka.apache.org>
> > > *Subject: *[EXTERNAL] Re: Health Check
> > >
> > >
> > >
> > > We've wired the Prometheus JMX exporter java-agent into our Kafka
> > > processes, we've then set up alerts in Prometheus for several key
> metrics
> > > to monitor cluster health (under replicated partitions/offline
> > partitions,
> > > request handler avg idle percent to measure load etc.) - and because
> the
> > > JMX exporter runs within the Kafka process, any sustained failure to
> > scrape
> > > also sets off an alert.
> > >
> > > Cheers,
> > >
> > > Liam Clarke
> > >
> > > On Thu, Dec 19, 2019 at 12:42 AM Miguel Silvestre <
> msilvestre@gmail.com>
> > > wrote:
> > >
> > > A simple tcp connection to kafka port (9092) should be enough no?
> > >
> > > --
> > > Miguel Silvestre
> >
> >
>
Re: [EXTERNAL] Re: Health Check
Posted by Alex Woolford <al...@woolford.io>.
Not sure if this is helpful, Tim.
I recently recorded a video that shows the gist of monitoring Kafka with
Prometheus: https://www.youtube.com/watch?v=nk3sk1LO7Bo
Cheers,
Alex
On Sat, Dec 21, 2019 at 12:13 PM Liam Clarke <li...@adscale.co.nz>
wrote:
> Nope, it's a third party project:
> https://github.com/prometheus/jmx_exporter
>
> We're using it with an 0.11 cluster.
>
> On Sat, 21 Dec. 2019, 1:52 am Sullivan, Tim, <TI...@homedepot.com>
> wrote:
>
> >
> > Wonderful, I assume that it is only in the latest release,
> > correct? I’m running 1.1.0 so I’m thinking I’d have to upgrade to get the
> > benefits?
> >
> >
> > Tim Sullivan
> >
> > Be Well
> >
> >
> >
> > Sr. Software Engineer
> > Supply Chain – IT | Data & Analytics
> > The Home Depot
> > 2250 Newmarket Parkway | Atlanta, GA 30339
> >
> > Work Cell 470-455-8346
> > Personal Cell 678-525-2583
> > Home Phone: 770-945-3315
> >
> >
> > *From: *Liam Clarke <li...@adscale.co.nz>
> > *Reply-To: *"users@kafka.apache.org" <us...@kafka.apache.org>
> > *Date: *Thursday, December 19, 2019 at 6:13 PM
> > *To: *"users@kafka.apache.org" <us...@kafka.apache.org>
> > *Subject: *[EXTERNAL] Re: Health Check
> >
> >
> >
> > We've wired the Prometheus JMX exporter java-agent into our Kafka
> > processes, we've then set up alerts in Prometheus for several key metrics
> > to monitor cluster health (under replicated partitions/offline
> partitions,
> > request handler avg idle percent to measure load etc.) - and because the
> > JMX exporter runs within the Kafka process, any sustained failure to
> scrape
> > also sets off an alert.
> >
> > Cheers,
> >
> > Liam Clarke
> >
> > On Thu, Dec 19, 2019 at 12:42 AM Miguel Silvestre <ms...@gmail.com>
> > wrote:
> >
> > A simple tcp connection to kafka port (9092) should be enough no?
> >
> > --
> > Miguel Silvestre
>
>
Re: [EXTERNAL] Re: Health Check
Posted by Liam Clarke <li...@adscale.co.nz>.
Nope, it's a third party project:
https://github.com/prometheus/jmx_exporter
We're using it with an 0.11 cluster.
On Sat, 21 Dec. 2019, 1:52 am Sullivan, Tim, <TI...@homedepot.com>
wrote:
>
>
> Wonderful, I assume that it is only in the latest release,
> correct? I’m running 1.1.0 so I’m thinking I’d have to upgrade to get the
> benefits?
>
>
>
>
>
> Tim Sullivan
>
> Be Well
>
>
>
> Sr. Software Engineer
>
> Supply Chain – IT | Data & Analytics
>
> The Home Depot
>
> 2250 Newmarket Parkway | Atlanta, GA 30339
>
>
>
> Work Cell 470-455-8346
>
> Personal Cell 678-525-2583
>
> Home Phone: 770-945-3315
>
>
>
> [image: signature_522277300]
>
>
>
>
>
>
>
> *From: *Liam Clarke <li...@adscale.co.nz>
> *Reply-To: *"users@kafka.apache.org" <us...@kafka.apache.org>
> *Date: *Thursday, December 19, 2019 at 6:13 PM
> *To: *"users@kafka.apache.org" <us...@kafka.apache.org>
> *Subject: *[EXTERNAL] Re: Health Check
>
>
>
> We've wired the Prometheus JMX exporter java-agent into our Kafka
>
> processes, we've then set up alerts in Prometheus for several key metrics
>
> to monitor cluster health (under replicated partitions/offline partitions,
>
> request handler avg idle percent to measure load etc.) - and because the
>
> JMX exporter runs within the Kafka process, any sustained failure to scrape
>
> also sets off an alert.
>
>
>
> Cheers,
>
>
>
> Liam Clarke
>
>
>
> On Thu, Dec 19, 2019 at 12:42 AM Miguel Silvestre <ms...@gmail.com>
>
> wrote:
>
>
>
> A simple tcp connection to kafka port (9092) should be enough no?
>
> --
>
> Miguel Silvestre
>
>
>
>
>
> On Tue, Dec 17, 2019 at 8:33 PM M. Manna <ma...@gmail.com> wrote:
>
>
>
> > Jai,
>
> >
>
> > On Tue, 17 Dec 2019 at 17:33, Jai Nanda Kumar <ja...@idexcel.com>
>
> > wrote:
>
> >
>
> > > Hi,
>
> > >
>
> > > How to perform a health check on a running Kafka server in AWS EC2
>
> > > server.
>
> > >
>
> >
>
> > SHouldn't this be part of your liveness probe? Or, are you trying to do
>
> > this adhoc (not how folks do it anyway)?
>
> >
>
> > >
>
> > > Thanks and Regards,
>
> > > A. Jai Nanda Kumar,
>
> > > Senior Software Engineer,
>
> > > 9066260799.
>
> > > Confidentiality Notice: This e-mail message, including any attachments,
>
> > is
>
> > > for the sole use of the intended recipient(s) and may contain
>
> > confidential
>
> > > and privileged information. Any unauthorized review, use, disclosure or
>
> > > distribution is prohibited. If you are not the intended recipient,
>
> please
>
> > > contact the sender by reply e-mail and destroy all copies of the
>
> original
>
> > > message.
>
> > >
>
> >
>
>
>
>
>
> ------------------------------
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>