You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Robin Yamaguchi <ro...@hasoffers.com> on 2014/04/29 01:49:55 UTC
Can't find certain mbeans
Greetings,
I'm working with Kafka 0.8.1. and setting up monitoring / metrics trending.
I've been exploring mbeans on my kafka broker, and unable to find the
following based on this documentation:
http://kafka.apache.org/081/documentation.html#monitoring
1.)
"kafka.log":name="LogFlushRateAndTimeMs",type="LogFlushStats"
I only see mbeans of *-LogEndOffset and *-NumLogSements for each topic
partition in the "kafka.log" domain.
2.)
"kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager"
I only see these mbeans:
#domain = "kafka.consumer":
"kafka.consumer":name="ReplicaFetcherThread-0-1-AllBrokersFetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
"kafka.consumer":name="ReplicaFetcherThread-0-1-AllBrokersFetchResponseSize",type="FetchRequestAndResponseMetrics"
"kafka.consumer":name="ReplicaFetcherThread-0-1-host_kafka_host01-port_13001-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
"kafka.consumer":name="ReplicaFetcherThread-0-1-host_kafka_host01-port_13001-FetchResponseSize",type="FetchRequestAndResponseMetrics"
"kafka.consumer":name="ReplicaFetcherThread-0-2-AllBrokersFetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
"kafka.consumer":name="ReplicaFetcherThread-0-2-AllBrokersFetchResponseSize",type="FetchRequestAndResponseMetrics"
"kafka.consumer":name="ReplicaFetcherThread-0-2-host_kafka_host02-port_13002-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
"kafka.consumer":name="ReplicaFetcherThread-0-2-host_kafka_host02-port_13002-FetchResponseSize",type="FetchRequestAndResponseMetrics"
Am I missing something?
Thank you,
Robin
Re: Can't find certain mbeans
Posted by ravi singh <rr...@gmail.com>.
These are certain Mbeans which will be available at consumer end.
Regards,
Ravi
On Mon, Jul 28, 2014 at 3:28 PM, 이재익 <ja...@navercorp.com> wrote:
> Hello,
>
>
> I have same problem. Do you have any update about for 2)?
>
> I'm using kafka 0.8.1.1 and replication factor is 2.
>
>
>
> In my case, I installed "Kafka Graphite Metrics Reporter", and try to
> monitor metrics through graphite.
>
> (https://github.com/damienclaveau/kafka-graphite)
>
>
>
> But following metric is not available in kafka.consumer only
> "ReplicaFetcherThread"* are available.
>
> "kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager"
>
>
>
> I like to do same thing that " bin/kafka-run-class.sh
> kafka.tools.ConsumerOffsetChecker" is doing through graphite.
>
>
>
>
>
> -----Original Message-----
> *From:* "Robin Yamaguchi"<ro...@hasoffers.com>
> *To:* <us...@kafka.apache.org>;
> *Cc:*
> *Sent:* 2014-04-30 (수) 03:26:15
> *Subject:* Re: Can't find certain mbeans
>
> For 2), do you have a topic with at least 2 replicas?
>
> Yes, all of our topics are configured with a replication factor of 3.
> Here's the output of --describe on one of them:
>
> Topic:QProcTopic4 PartitionCount:16 ReplicationFactor:3 Configs:
> Topic: QProcTopic4 Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
> Topic: QProcTopic4 Partition: 1 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
> Topic: QProcTopic4 Partition: 2 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
> Topic: QProcTopic4 Partition: 3 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
> Topic: QProcTopic4 Partition: 4 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
> Topic: QProcTopic4 Partition: 5 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
> Topic: QProcTopic4 Partition: 6 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
> Topic: QProcTopic4 Partition: 7 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
> Topic: QProcTopic4 Partition: 8 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
> Topic: QProcTopic4 Partition: 9 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
> Topic: QProcTopic4 Partition: 10 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
> Topic: QProcTopic4 Partition: 11 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
> Topic: QProcTopic4 Partition: 12 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
> Topic: QProcTopic4 Partition: 13 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
> Topic: QProcTopic4 Partition: 14 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
> Topic: QProcTopic4 Partition: 15 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
>
> For 1), was the flush interval reached?
>
> Yes, I believe so. log.flush.interval is set to the default 10000. After
> passing 25K messages through the broker, I still do not see those mbeans.
>
> Thank you for your help with this. Let me know if there's any more
> information I can provide. We will be upgrading our clusters to 0.8.1.1 at
> some point this week.
>
> thanks,
> Robin
>
>
> On Mon, Apr 28, 2014 at 8:56 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > For 2), do you have a topic with at least 2 replicas? The metrics will
> only
> > show up if the broker has a follower replica following the leader replica
> > on another broker.
> >
> > For 1), was the flush interval reached?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Apr 28, 2014 at 4:49 PM, Robin Yamaguchi <robin>@hasoffers.com
> > >wrote:
> >
> > > Greetings,
> > >
> > > I'm working with Kafka 0.8.1. and setting up monitoring / metrics
> > trending.
> > > I've been exploring mbeans on my kafka broker, and unable to find the
> > > following based on this documentation:
> > > http://kafka.apache.org/081/documentation.html#monitoring
> > >
> > > 1.)
> > > "kafka.log":name="LogFlushRateAndTimeMs",type="LogFlushStats"
> > >
> > > I only see mbeans of *-LogEndOffset and *-NumLogSements for each topic
> > > partition in the "kafka.log" domain.
> > >
> > > 2.)
> > > "kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager"
> > >
> > > I only see these mbeans:
> > >
> > > #domain = "kafka.consumer":
> > >
> > >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-AllBrokersFetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> > >
> > >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-AllBrokersFetchResponseSize",type="FetchRequestAndResponseMetrics"
> > >
> > >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-host_kafka_host01-port_13001-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> > >
> > >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-host_kafka_host01-port_13001-FetchResponseSize",type="FetchRequestAndResponseMetrics"
> > >
> > >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-AllBrokersFetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> > >
> > >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-AllBrokersFetchResponseSize",type="FetchRequestAndResponseMetrics"
> > >
> > >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-host_kafka_host02-port_13002-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> > >
> > >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-host_kafka_host02-port_13002-FetchResponseSize",type="FetchRequestAndResponseMetrics"
> > >
> > > Am I missing something?
> > >
> > > Thank you,
> > > Robin
> > >
> >
>
--
*Regards,*
*Ravi*
Re: Can't find certain mbeans
Posted by 이재익 <ja...@navercorp.com>.
Hello,
I have same problem. Do you have any update about for 2)?
I'm using kafka 0.8.1.1 and replication factor is 2.
In my case, I installed "Kafka Graphite Metrics Reporter", and try to monitor metrics through graphite.
(https://github.com/damienclaveau/kafka-graphite)
But following metric is not available in kafka.consumer only "ReplicaFetcherThread"* are available.
"kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager"
I like to do same thing that " bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker" is doing through graphite.
-----Original Message-----
From: "Robin Yamaguchi"<robin@hasoffers.com>
To: <users@kafka.apache.org>;
Cc:
Sent: 2014-04-30 (수) 03:26:15
Subject: Re: Can't find certain mbeans
For 2), do you have a topic with at least 2 replicas?
Yes, all of our topics are configured with a replication factor of 3.
Here's the output of --describe on one of them:
Topic:QProcTopic4 PartitionCount:16 ReplicationFactor:3 Configs:
Topic: QProcTopic4 Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: QProcTopic4 Partition: 1 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: QProcTopic4 Partition: 2 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: QProcTopic4 Partition: 3 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
Topic: QProcTopic4 Partition: 4 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: QProcTopic4 Partition: 5 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: QProcTopic4 Partition: 6 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: QProcTopic4 Partition: 7 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: QProcTopic4 Partition: 8 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: QProcTopic4 Partition: 9 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
Topic: QProcTopic4 Partition: 10 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: QProcTopic4 Partition: 11 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: QProcTopic4 Partition: 12 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: QProcTopic4 Partition: 13 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: QProcTopic4 Partition: 14 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: QProcTopic4 Partition: 15 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
For 1), was the flush interval reached?
Yes, I believe so. log.flush.interval is set to the default 10000. After
passing 25K messages through the broker, I still do not see those mbeans.
Thank you for your help with this. Let me know if there's any more
information I can provide. We will be upgrading our clusters to 0.8.1.1 at
some point this week.
thanks,
Robin
On Mon, Apr 28, 2014 at 8:56 PM, Jun Rao <junrao>@gmail.com> wrote:
> For 2), do you have a topic with at least 2 replicas? The metrics will only
> show up if the broker has a follower replica following the leader replica
> on another broker.
>
> For 1), was the flush interval reached?
>
> Thanks,
>
> Jun
>
>
> On Mon, Apr 28, 2014 at 4:49 PM, Robin Yamaguchi <robin>@hasoffers.com
> >wrote:
>
> > Greetings,
> >
> > I'm working with Kafka 0.8.1. and setting up monitoring / metrics
> trending.
> > I've been exploring mbeans on my kafka broker, and unable to find the
> > following based on this documentation:
> > http://kafka.apache.org/081/documentation.html#monitoring
> >
> > 1.)
> > "kafka.log":name="LogFlushRateAndTimeMs",type="LogFlushStats"
> >
> > I only see mbeans of *-LogEndOffset and *-NumLogSements for each topic
> > partition in the "kafka.log" domain.
> >
> > 2.)
> > "kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager"
> >
> > I only see these mbeans:
> >
> > #domain = "kafka.consumer":
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-AllBrokersFetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-AllBrokersFetchResponseSize",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-host_kafka_host01-port_13001-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-host_kafka_host01-port_13001-FetchResponseSize",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-AllBrokersFetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-AllBrokersFetchResponseSize",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-host_kafka_host02-port_13002-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-host_kafka_host02-port_13002-FetchResponseSize",type="FetchRequestAndResponseMetrics"
> >
> > Am I missing something?
> >
> > Thank you,
> > Robin
> >
>
Re: Can't find certain mbeans
Posted by Robin Yamaguchi <ro...@hasoffers.com>.
For 2), do you have a topic with at least 2 replicas?
Yes, all of our topics are configured with a replication factor of 3.
Here's the output of --describe on one of them:
Topic:QProcTopic4 PartitionCount:16 ReplicationFactor:3 Configs:
Topic: QProcTopic4 Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: QProcTopic4 Partition: 1 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: QProcTopic4 Partition: 2 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: QProcTopic4 Partition: 3 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
Topic: QProcTopic4 Partition: 4 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: QProcTopic4 Partition: 5 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: QProcTopic4 Partition: 6 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: QProcTopic4 Partition: 7 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: QProcTopic4 Partition: 8 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: QProcTopic4 Partition: 9 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
Topic: QProcTopic4 Partition: 10 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: QProcTopic4 Partition: 11 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: QProcTopic4 Partition: 12 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: QProcTopic4 Partition: 13 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: QProcTopic4 Partition: 14 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: QProcTopic4 Partition: 15 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
For 1), was the flush interval reached?
Yes, I believe so. log.flush.interval is set to the default 10000. After
passing 25K messages through the broker, I still do not see those mbeans.
Thank you for your help with this. Let me know if there's any more
information I can provide. We will be upgrading our clusters to 0.8.1.1 at
some point this week.
thanks,
Robin
On Mon, Apr 28, 2014 at 8:56 PM, Jun Rao <ju...@gmail.com> wrote:
> For 2), do you have a topic with at least 2 replicas? The metrics will only
> show up if the broker has a follower replica following the leader replica
> on another broker.
>
> For 1), was the flush interval reached?
>
> Thanks,
>
> Jun
>
>
> On Mon, Apr 28, 2014 at 4:49 PM, Robin Yamaguchi <robin@hasoffers.com
> >wrote:
>
> > Greetings,
> >
> > I'm working with Kafka 0.8.1. and setting up monitoring / metrics
> trending.
> > I've been exploring mbeans on my kafka broker, and unable to find the
> > following based on this documentation:
> > http://kafka.apache.org/081/documentation.html#monitoring
> >
> > 1.)
> > "kafka.log":name="LogFlushRateAndTimeMs",type="LogFlushStats"
> >
> > I only see mbeans of *-LogEndOffset and *-NumLogSements for each topic
> > partition in the "kafka.log" domain.
> >
> > 2.)
> > "kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager"
> >
> > I only see these mbeans:
> >
> > #domain = "kafka.consumer":
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-AllBrokersFetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-AllBrokersFetchResponseSize",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-host_kafka_host01-port_13001-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-1-host_kafka_host01-port_13001-FetchResponseSize",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-AllBrokersFetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-AllBrokersFetchResponseSize",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-host_kafka_host02-port_13002-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> >
> >
> "kafka.consumer":name="ReplicaFetcherThread-0-2-host_kafka_host02-port_13002-FetchResponseSize",type="FetchRequestAndResponseMetrics"
> >
> > Am I missing something?
> >
> > Thank you,
> > Robin
> >
>
Re: Can't find certain mbeans
Posted by Jun Rao <ju...@gmail.com>.
For 2), do you have a topic with at least 2 replicas? The metrics will only
show up if the broker has a follower replica following the leader replica
on another broker.
For 1), was the flush interval reached?
Thanks,
Jun
On Mon, Apr 28, 2014 at 4:49 PM, Robin Yamaguchi <ro...@hasoffers.com>wrote:
> Greetings,
>
> I'm working with Kafka 0.8.1. and setting up monitoring / metrics trending.
> I've been exploring mbeans on my kafka broker, and unable to find the
> following based on this documentation:
> http://kafka.apache.org/081/documentation.html#monitoring
>
> 1.)
> "kafka.log":name="LogFlushRateAndTimeMs",type="LogFlushStats"
>
> I only see mbeans of *-LogEndOffset and *-NumLogSements for each topic
> partition in the "kafka.log" domain.
>
> 2.)
> "kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager"
>
> I only see these mbeans:
>
> #domain = "kafka.consumer":
>
> "kafka.consumer":name="ReplicaFetcherThread-0-1-AllBrokersFetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
>
> "kafka.consumer":name="ReplicaFetcherThread-0-1-AllBrokersFetchResponseSize",type="FetchRequestAndResponseMetrics"
>
> "kafka.consumer":name="ReplicaFetcherThread-0-1-host_kafka_host01-port_13001-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
>
> "kafka.consumer":name="ReplicaFetcherThread-0-1-host_kafka_host01-port_13001-FetchResponseSize",type="FetchRequestAndResponseMetrics"
>
> "kafka.consumer":name="ReplicaFetcherThread-0-2-AllBrokersFetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
>
> "kafka.consumer":name="ReplicaFetcherThread-0-2-AllBrokersFetchResponseSize",type="FetchRequestAndResponseMetrics"
>
> "kafka.consumer":name="ReplicaFetcherThread-0-2-host_kafka_host02-port_13002-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
>
> "kafka.consumer":name="ReplicaFetcherThread-0-2-host_kafka_host02-port_13002-FetchResponseSize",type="FetchRequestAndResponseMetrics"
>
> Am I missing something?
>
> Thank you,
> Robin
>