You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Koert Kuipers <ko...@tresata.com> on 2015/02/09 23:48:55 UTC
ping kafka server
i would like to be able to ping kafka servers from nagios to confirm they
are alive. since kafka servers dont run a http server (web ui) i am not
sure how to do this.
is it safe to establish a "test" tcp connection (so connect and immediately
disconnect using telnet or netstat or something like that) to the kafka
server on port 9092 to confirm its alive?
thanks
Re: ping kafka server
Posted by to...@borked.ca.
Hi gwen,
Can you share how you do these end to end latency tests? I am more sysadmin than coder and have wanted to get something like that going for my kafka clusters. I'd love more details about how you do it, and how you monitor the results.
Thanks!
Sent from my BlackBerry 10 smartphone on the TELUS network.
Original Message
From: Gwen Shapira
Sent: Monday, February 9, 2015 5:55 PM
To: users@kafka.apache.org
Reply To: users@kafka.apache.org
Subject: Re: ping kafka server
It's safe.
Just note that if you send Kafka anything it does not like, it will close
the connection on you. This is intentional and doesn't signal an issue with
Kafka.
Not sure if Nagios does this, but I like "canary" tests - produce a message
with timestamp every X seconds and have a monitor that consumes the
messages and check timestamps. This way you get both end-to-end monitoring
and latency alerts. What could be more fun? :)
Gwen
On Mon, Feb 9, 2015 at 2:48 PM, Koert Kuipers <ko...@tresata.com> wrote:
> i would like to be able to ping kafka servers from nagios to confirm they
> are alive. since kafka servers dont run a http server (web ui) i am not
> sure how to do this.
>
> is it safe to establish a "test" tcp connection (so connect and immediately
> disconnect using telnet or netstat or something like that) to the kafka
> server on port 9092 to confirm its alive?
>
> thanks
>
Re: ping kafka server
Posted by Gwen Shapira <gs...@cloudera.com>.
It's safe.
Just note that if you send Kafka anything it does not like, it will close
the connection on you. This is intentional and doesn't signal an issue with
Kafka.
Not sure if Nagios does this, but I like "canary" tests - produce a message
with timestamp every X seconds and have a monitor that consumes the
messages and check timestamps. This way you get both end-to-end monitoring
and latency alerts. What could be more fun? :)
Gwen
On Mon, Feb 9, 2015 at 2:48 PM, Koert Kuipers <ko...@tresata.com> wrote:
> i would like to be able to ping kafka servers from nagios to confirm they
> are alive. since kafka servers dont run a http server (web ui) i am not
> sure how to do this.
>
> is it safe to establish a "test" tcp connection (so connect and immediately
> disconnect using telnet or netstat or something like that) to the kafka
> server on port 9092 to confirm its alive?
>
> thanks
>
Re: ping kafka server
Posted by Joe Stein <jo...@stealth.ly>.
One good canary is to have a topic you loop through every partition and
write a timestamp on every broker for n in 1..P; do; echo
$(datetime)>kafkacat -b host:port -t ops -p n done and trap any errors and
output accordingly.
You can get fancy and start validating reading and stuff but it is a canary
after all.
~ Joestein
On Tue, Feb 10, 2015 at 12:17 PM, Magnus Edenhill <ma...@edenhill.se>
wrote:
> Relying on just the TCP connection getting established seems a bit poor,
> the easiest non-intrusive approach is probably to query the broker for
> metadata,
> e.g.: kafkacat -b mybroker -L
>
>
> 2015-02-10 1:47 GMT+01:00 Koert Kuipers <ko...@tresata.com>:
>
> > a simple nagios check_tcp works fine. as gwen indicated kafka closes the
> > connection on me, but this is (supposedly) harmless. i see in server
> logs:
> > [2015-02-09 19:39:17,069] INFO Closing socket connection to /
> 192.168.1.31.
> > (kafka.network.Processor)
> >
> >
> > On Mon, Feb 9, 2015 at 6:06 PM, Scott Clasen <sc...@heroku.com> wrote:
> >
> > > I have used nagios in this manner with kafaka before and worked fine.
> > >
> > > On Mon, Feb 9, 2015 at 2:48 PM, Koert Kuipers <ko...@tresata.com>
> wrote:
> > >
> > > > i would like to be able to ping kafka servers from nagios to confirm
> > they
> > > > are alive. since kafka servers dont run a http server (web ui) i am
> not
> > > > sure how to do this.
> > > >
> > > > is it safe to establish a "test" tcp connection (so connect and
> > > immediately
> > > > disconnect using telnet or netstat or something like that) to the
> kafka
> > > > server on port 9092 to confirm its alive?
> > > >
> > > > thanks
> > > >
> > >
> >
>
Re: ping kafka server
Posted by Magnus Edenhill <ma...@edenhill.se>.
Relying on just the TCP connection getting established seems a bit poor,
the easiest non-intrusive approach is probably to query the broker for
metadata,
e.g.: kafkacat -b mybroker -L
2015-02-10 1:47 GMT+01:00 Koert Kuipers <ko...@tresata.com>:
> a simple nagios check_tcp works fine. as gwen indicated kafka closes the
> connection on me, but this is (supposedly) harmless. i see in server logs:
> [2015-02-09 19:39:17,069] INFO Closing socket connection to /192.168.1.31.
> (kafka.network.Processor)
>
>
> On Mon, Feb 9, 2015 at 6:06 PM, Scott Clasen <sc...@heroku.com> wrote:
>
> > I have used nagios in this manner with kafaka before and worked fine.
> >
> > On Mon, Feb 9, 2015 at 2:48 PM, Koert Kuipers <ko...@tresata.com> wrote:
> >
> > > i would like to be able to ping kafka servers from nagios to confirm
> they
> > > are alive. since kafka servers dont run a http server (web ui) i am not
> > > sure how to do this.
> > >
> > > is it safe to establish a "test" tcp connection (so connect and
> > immediately
> > > disconnect using telnet or netstat or something like that) to the kafka
> > > server on port 9092 to confirm its alive?
> > >
> > > thanks
> > >
> >
>
Re: ping kafka server
Posted by Koert Kuipers <ko...@tresata.com>.
a simple nagios check_tcp works fine. as gwen indicated kafka closes the
connection on me, but this is (supposedly) harmless. i see in server logs:
[2015-02-09 19:39:17,069] INFO Closing socket connection to /192.168.1.31.
(kafka.network.Processor)
On Mon, Feb 9, 2015 at 6:06 PM, Scott Clasen <sc...@heroku.com> wrote:
> I have used nagios in this manner with kafaka before and worked fine.
>
> On Mon, Feb 9, 2015 at 2:48 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
> > i would like to be able to ping kafka servers from nagios to confirm they
> > are alive. since kafka servers dont run a http server (web ui) i am not
> > sure how to do this.
> >
> > is it safe to establish a "test" tcp connection (so connect and
> immediately
> > disconnect using telnet or netstat or something like that) to the kafka
> > server on port 9092 to confirm its alive?
> >
> > thanks
> >
>
Re: ping kafka server
Posted by Scott Clasen <sc...@heroku.com>.
I have used nagios in this manner with kafaka before and worked fine.
On Mon, Feb 9, 2015 at 2:48 PM, Koert Kuipers <ko...@tresata.com> wrote:
> i would like to be able to ping kafka servers from nagios to confirm they
> are alive. since kafka servers dont run a http server (web ui) i am not
> sure how to do this.
>
> is it safe to establish a "test" tcp connection (so connect and immediately
> disconnect using telnet or netstat or something like that) to the kafka
> server on port 9092 to confirm its alive?
>
> thanks
>