You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Alex Objelean <al...@gmail.com> on 2014/10/16 12:09:34 UTC
Monitoring connection with kafka client
Hi,
I'm trying to monitor the kafka connection on the consumer side. In other
words, if the broker cluster is unavailable (or zookeer dies), I would like
to know about that problem as soon as possible.
Unfortunately, I didn't find anything useful to achieve that when using
kafka library.
Are there any suggestions about how to fix this issue?
Thanks,
Alex
Re: Monitoring connection with kafka client
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
We use our own SPM <http://sematext.com/spm/> to monitor our Kafka brokers,
producers, and consumers (and ZK) and have alerts and anomaly detection on
several key Kafka and ZK metrics. When things break around ZK and/or
Kafka..... we find out pretty quickly because a lot of metrics suddenly
spike or drop :) In other words, if I understand your question correctly,
I'd suggest using a monitoring tool to detect things like unavailable Kafka
brokers or ZK nodes instead of manually watching connections with a Kafka
client...
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Thu, Oct 16, 2014 at 6:09 AM, Alex Objelean <al...@gmail.com>
wrote:
> Hi,
>
> I'm trying to monitor the kafka connection on the consumer side. In other
> words, if the broker cluster is unavailable (or zookeer dies), I would like
> to know about that problem as soon as possible.
> Unfortunately, I didn't find anything useful to achieve that when using
> kafka library.
> Are there any suggestions about how to fix this issue?
>
> Thanks,
> Alex
>
Re: Monitoring connection with kafka client
Posted by Neha Narkhede <ne...@gmail.com>.
If you want to know if the Kafka and zookeeper cluster is healthy or not,
you'd want to monitor the cluster directly. Here are pointers for
monitoring the Kafka brokers -
http://kafka.apache.org/documentation.html#monitoring
Thanks,
Neha
On Thu, Oct 16, 2014 at 3:09 AM, Alex Objelean <al...@gmail.com>
wrote:
> Hi,
>
> I'm trying to monitor the kafka connection on the consumer side. In other
> words, if the broker cluster is unavailable (or zookeer dies), I would like
> to know about that problem as soon as possible.
> Unfortunately, I didn't find anything useful to achieve that when using
> kafka library.
> Are there any suggestions about how to fix this issue?
>
> Thanks,
> Alex
>
Re: Monitoring connection with kafka client
Posted by Harsha <ka...@harsha.io>.
Alex,
In the above mentioned case I think monitoring the consumer lag
will be helpful
"kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager".
Another approach I used before was to use metrics library to send
consumer stats to ganglia and use nagios alert on those stats, If the
consumed messages drops below certain threshold .
-Harsha
On Fri, Oct 17, 2014, at 01:08 AM, Alex Objelean wrote:
> @Neha Narkede
>
> Though monitoring the health of Kafka & Zookeeper clusters directly is
> useful, it might not be enough.
> Consider the following scenario:
>
> You have a client consuming messages. Zookeeper and kafka nodes are
> healthy. The monitor says everything is ok. For some reason, a connection
> between client and zookeeper fails (new firewall rule or connectivity
> issues between different data centers, doesn't really matter). Since the
> zookeeper and kafka nodes are still healthy, the monitor is still
> reporting
> OK. The client retries to reconnect indefinitely, but unless a human is
> watching the logs, the problem is not visible by anybody.
>
> Having a listener which is invoked when the connection state is changed
> is
> something that is provided by other messaging frameworks. Having this
> feature added to kafka would be a nice addition.
>
> Thanks,
> Alex
>
> On 17 October 2014 10:55, Alex Objelean <al...@gmail.com> wrote:
>
> > @Otis thanks for your answer.
> >
> > Of course it is not about manually watching connection. We already have
> > monitoring tools in place. But I would expect the kafka client to provide
> > means for get notified about potential connectivity issues.
> > Actual behavior is this: the client retries to reconnect indefinitely (I
> > see errors in logs). What I need, is a kind of listener which would be
> > invoked when the connection state change. In other words, once the client
> > notice the problem and is trying to reconnect, our monitoring tools would
> > reflect the connection state as soon as the problem appears.
> >
> > Thanks,
> > Alex
> >
> >
> >
> > On 16 October 2014 13:09, Alex Objelean <al...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I'm trying to monitor the kafka connection on the consumer side. In other
> >> words, if the broker cluster is unavailable (or zookeer dies), I would like
> >> to know about that problem as soon as possible.
> >> Unfortunately, I didn't find anything useful to achieve that when using
> >> kafka library.
> >> Are there any suggestions about how to fix this issue?
> >>
> >> Thanks,
> >> Alex
> >>
> >
> >
Re: Monitoring connection with kafka client
Posted by Alex Objelean <al...@gmail.com>.
@Neha Narkede
Though monitoring the health of Kafka & Zookeeper clusters directly is
useful, it might not be enough.
Consider the following scenario:
You have a client consuming messages. Zookeeper and kafka nodes are
healthy. The monitor says everything is ok. For some reason, a connection
between client and zookeeper fails (new firewall rule or connectivity
issues between different data centers, doesn't really matter). Since the
zookeeper and kafka nodes are still healthy, the monitor is still reporting
OK. The client retries to reconnect indefinitely, but unless a human is
watching the logs, the problem is not visible by anybody.
Having a listener which is invoked when the connection state is changed is
something that is provided by other messaging frameworks. Having this
feature added to kafka would be a nice addition.
Thanks,
Alex
On 17 October 2014 10:55, Alex Objelean <al...@gmail.com> wrote:
> @Otis thanks for your answer.
>
> Of course it is not about manually watching connection. We already have
> monitoring tools in place. But I would expect the kafka client to provide
> means for get notified about potential connectivity issues.
> Actual behavior is this: the client retries to reconnect indefinitely (I
> see errors in logs). What I need, is a kind of listener which would be
> invoked when the connection state change. In other words, once the client
> notice the problem and is trying to reconnect, our monitoring tools would
> reflect the connection state as soon as the problem appears.
>
> Thanks,
> Alex
>
>
>
> On 16 October 2014 13:09, Alex Objelean <al...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm trying to monitor the kafka connection on the consumer side. In other
>> words, if the broker cluster is unavailable (or zookeer dies), I would like
>> to know about that problem as soon as possible.
>> Unfortunately, I didn't find anything useful to achieve that when using
>> kafka library.
>> Are there any suggestions about how to fix this issue?
>>
>> Thanks,
>> Alex
>>
>
>
Re: Monitoring connection with kafka client
Posted by Alex Objelean <al...@gmail.com>.
@Otis thanks for your answer.
Of course it is not about manually watching connection. We already have
monitoring tools in place. But I would expect the kafka client to provide
means for get notified about potential connectivity issues.
Actual behavior is this: the client retries to reconnect indefinitely (I
see errors in logs). What I need, is a kind of listener which would be
invoked when the connection state change. In other words, once the client
notice the problem and is trying to reconnect, our monitoring tools would
reflect the connection state as soon as the problem appears.
Thanks,
Alex
On 16 October 2014 13:09, Alex Objelean <al...@gmail.com> wrote:
> Hi,
>
> I'm trying to monitor the kafka connection on the consumer side. In other
> words, if the broker cluster is unavailable (or zookeer dies), I would like
> to know about that problem as soon as possible.
> Unfortunately, I didn't find anything useful to achieve that when using
> kafka library.
> Are there any suggestions about how to fix this issue?
>
> Thanks,
> Alex
>