You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Otis Gospodnetić <ot...@gmail.com> on 2015/09/12 03:25:13 UTC
Re: Zookeeper jmx monitoring for kafka
Hi Prabhjot,
Short answer: yes
I used to think ZK was so super stable that it was one of those things that
don't require any management, but on a few occasions I witnessed complex
distributed applications nearly fall apart because of issues with ZK. We
use our own SPM for ZooKeeper to monitor all our ZK nodes. I'm not a ZK
expert, but metrics like Request Latency, Watch Count, Outstanding Requests
seem important.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Aug 18, 2015 at 9:11 AM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hi,
>
> Zookeeper monitoring - I was going through the 'Monitoring' section in
> http://kafka.apache.org/coding-guide.html
>
> I have setup Ganglia to monitor all the stats inocming from kafka jmx port
> But, in addition, there are a bunch of stats that Zookeeper also exposes in
> its jmx port.
> They are documented here:
> https://zookeeper.apache.org/doc/r3.3.4/zookeeperJMX.html
>
> Is it worthwhile setting up Ganglia for Zookeeper along with Kafka ?
> If yes, in your experience, what are the essential stats to look out for
> (especially in relation to its use with Kafka)
>
> Looking forward to your response.
>
> Thanks,
> prabcs
>
Re: Zookeeper jmx monitoring for kafka
Posted by Edward Ribeiro <ed...@gmail.com>.
Fyi regarding disk space utilization, I submitted a patch to track ZK log
and snapshot directory sizes: ZOOKEEPER-1423
Best,
Eddie
Em 12/09/2015 11:44, "Flavio Junqueira" <fp...@apache.org> escreveu:
> We do support jmx extensively, and I believe you can use it Ganglia. As
> for metrics, things that come to mind now on top of what Otis suggested are:
>
> - Frequency of leader election (if they happen too frequently, then that's
> going to affect your availability and probably an indication of something
> wrong)
> - Disk space utilization
> - Number of sessions open per server
> - Number of znodes, database size and such
>
> You can also use four-letter words to get some info out of servers. Check
> the documentation for the ones available.
>
> -Flavio
>
> > On 12 Sep 2015, at 02:25, Otis Gospodnetić <ot...@gmail.com>
> wrote:
> >
> > Hi Prabhjot,
> >
> > Short answer: yes
> > I used to think ZK was so super stable that it was one of those things
> that
> > don't require any management, but on a few occasions I witnessed complex
> > distributed applications nearly fall apart because of issues with ZK. We
> > use our own SPM for ZooKeeper to monitor all our ZK nodes. I'm not a ZK
> > expert, but metrics like Request Latency, Watch Count, Outstanding
> Requests
> > seem important.
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Tue, Aug 18, 2015 at 9:11 AM, Prabhjot Bharaj <pr...@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Zookeeper monitoring - I was going through the 'Monitoring' section in
> >> http://kafka.apache.org/coding-guide.html
> >>
> >> I have setup Ganglia to monitor all the stats inocming from kafka jmx
> port
> >> But, in addition, there are a bunch of stats that Zookeeper also
> exposes in
> >> its jmx port.
> >> They are documented here:
> >> https://zookeeper.apache.org/doc/r3.3.4/zookeeperJMX.html
> >>
> >> Is it worthwhile setting up Ganglia for Zookeeper along with Kafka ?
> >> If yes, in your experience, what are the essential stats to look out for
> >> (especially in relation to its use with Kafka)
> >>
> >> Looking forward to your response.
> >>
> >> Thanks,
> >> prabcs
> >>
>
>
Re: Zookeeper jmx monitoring for kafka
Posted by Edward Ribeiro <ed...@gmail.com>.
Fyi regarding disk space utilization, I submitted a patch to track ZK log
and snapshot directory sizes: ZOOKEEPER-1423
Best,
Eddie
Em 12/09/2015 11:44, "Flavio Junqueira" <fp...@apache.org> escreveu:
> We do support jmx extensively, and I believe you can use it Ganglia. As
> for metrics, things that come to mind now on top of what Otis suggested are:
>
> - Frequency of leader election (if they happen too frequently, then that's
> going to affect your availability and probably an indication of something
> wrong)
> - Disk space utilization
> - Number of sessions open per server
> - Number of znodes, database size and such
>
> You can also use four-letter words to get some info out of servers. Check
> the documentation for the ones available.
>
> -Flavio
>
> > On 12 Sep 2015, at 02:25, Otis Gospodnetić <ot...@gmail.com>
> wrote:
> >
> > Hi Prabhjot,
> >
> > Short answer: yes
> > I used to think ZK was so super stable that it was one of those things
> that
> > don't require any management, but on a few occasions I witnessed complex
> > distributed applications nearly fall apart because of issues with ZK. We
> > use our own SPM for ZooKeeper to monitor all our ZK nodes. I'm not a ZK
> > expert, but metrics like Request Latency, Watch Count, Outstanding
> Requests
> > seem important.
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Tue, Aug 18, 2015 at 9:11 AM, Prabhjot Bharaj <pr...@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Zookeeper monitoring - I was going through the 'Monitoring' section in
> >> http://kafka.apache.org/coding-guide.html
> >>
> >> I have setup Ganglia to monitor all the stats inocming from kafka jmx
> port
> >> But, in addition, there are a bunch of stats that Zookeeper also
> exposes in
> >> its jmx port.
> >> They are documented here:
> >> https://zookeeper.apache.org/doc/r3.3.4/zookeeperJMX.html
> >>
> >> Is it worthwhile setting up Ganglia for Zookeeper along with Kafka ?
> >> If yes, in your experience, what are the essential stats to look out for
> >> (especially in relation to its use with Kafka)
> >>
> >> Looking forward to your response.
> >>
> >> Thanks,
> >> prabcs
> >>
>
>
Re: Zookeeper jmx monitoring for kafka
Posted by Flavio Junqueira <fp...@apache.org>.
We do support jmx extensively, and I believe you can use it Ganglia. As for metrics, things that come to mind now on top of what Otis suggested are:
- Frequency of leader election (if they happen too frequently, then that's going to affect your availability and probably an indication of something wrong)
- Disk space utilization
- Number of sessions open per server
- Number of znodes, database size and such
You can also use four-letter words to get some info out of servers. Check the documentation for the ones available.
-Flavio
> On 12 Sep 2015, at 02:25, Otis Gospodnetić <ot...@gmail.com> wrote:
>
> Hi Prabhjot,
>
> Short answer: yes
> I used to think ZK was so super stable that it was one of those things that
> don't require any management, but on a few occasions I witnessed complex
> distributed applications nearly fall apart because of issues with ZK. We
> use our own SPM for ZooKeeper to monitor all our ZK nodes. I'm not a ZK
> expert, but metrics like Request Latency, Watch Count, Outstanding Requests
> seem important.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Tue, Aug 18, 2015 at 9:11 AM, Prabhjot Bharaj <pr...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Zookeeper monitoring - I was going through the 'Monitoring' section in
>> http://kafka.apache.org/coding-guide.html
>>
>> I have setup Ganglia to monitor all the stats inocming from kafka jmx port
>> But, in addition, there are a bunch of stats that Zookeeper also exposes in
>> its jmx port.
>> They are documented here:
>> https://zookeeper.apache.org/doc/r3.3.4/zookeeperJMX.html
>>
>> Is it worthwhile setting up Ganglia for Zookeeper along with Kafka ?
>> If yes, in your experience, what are the essential stats to look out for
>> (especially in relation to its use with Kafka)
>>
>> Looking forward to your response.
>>
>> Thanks,
>> prabcs
>>
Re: Zookeeper jmx monitoring for kafka
Posted by Flavio Junqueira <fp...@apache.org>.
We do support jmx extensively, and I believe you can use it Ganglia. As for metrics, things that come to mind now on top of what Otis suggested are:
- Frequency of leader election (if they happen too frequently, then that's going to affect your availability and probably an indication of something wrong)
- Disk space utilization
- Number of sessions open per server
- Number of znodes, database size and such
You can also use four-letter words to get some info out of servers. Check the documentation for the ones available.
-Flavio
> On 12 Sep 2015, at 02:25, Otis Gospodnetić <ot...@gmail.com> wrote:
>
> Hi Prabhjot,
>
> Short answer: yes
> I used to think ZK was so super stable that it was one of those things that
> don't require any management, but on a few occasions I witnessed complex
> distributed applications nearly fall apart because of issues with ZK. We
> use our own SPM for ZooKeeper to monitor all our ZK nodes. I'm not a ZK
> expert, but metrics like Request Latency, Watch Count, Outstanding Requests
> seem important.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Tue, Aug 18, 2015 at 9:11 AM, Prabhjot Bharaj <pr...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Zookeeper monitoring - I was going through the 'Monitoring' section in
>> http://kafka.apache.org/coding-guide.html
>>
>> I have setup Ganglia to monitor all the stats inocming from kafka jmx port
>> But, in addition, there are a bunch of stats that Zookeeper also exposes in
>> its jmx port.
>> They are documented here:
>> https://zookeeper.apache.org/doc/r3.3.4/zookeeperJMX.html
>>
>> Is it worthwhile setting up Ganglia for Zookeeper along with Kafka ?
>> If yes, in your experience, what are the essential stats to look out for
>> (especially in relation to its use with Kafka)
>>
>> Looking forward to your response.
>>
>> Thanks,
>> prabcs
>>