Re: Issues with metrics collection

[Joel Koshy <jj...@gmail.com>]

On Thu, Jul 31, 2014 at 11:09:03AM -0700, Kiran Nagasubramanian wrote:
> Thanks for the response Jun and Otis.
> 
> Jun, you're right. Only the leaders are reporting. One of the brokers is
> not a leader for any of the partitions. Found the following info on the
> documentation page:
> 
> leader is the node responsible for all reads and writes for the given
> partition. Each node will be the leader for a *randomly* selected portion
> of the partitions.
> 
> I've been was under an assumption that the leadership for partitions is
> distributed across all the brokers. *Is there a way to make all the brokers
> take leadership as much evenly as possible?*

It is distributed, although it may not be an optimal distribution.
i.e., as long as you have more than just a handful of topics the
leader counts across the brokers should be relatively even. If you do
a rolling bounce of the cluster the leader counts can get skewed in
which case you can run a preferred replica leader election operation:
http://kafka.apache.org/documentation.html#basic_ops_leader_balancing

> 
> We're using a custom app for metrics collection.
> 
> Otis, we're using kafka_2.9.2-0.8.1
> 
> Thanks.
> 
> 
> 
> On Thu, Jul 31, 2014 at 6:26 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
> 
> > Hi,
> >
> > May also want to share version of Kafka.  We have SPM agents for Kafka
> > 0.7.x and 0.8.x and we haven't seen that.
> >
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> >
> > On Wed, Jul 30, 2014 at 10:12 PM, Kiran Nagasubramanian <
> > nkiran87@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > We're collecting the JMX metrics from the Kafka brokers. We're seeing a
> > > couple of issues. Could someone please throw some light if you've come
> > > across something similar?
> > >
> > > 1) We have a 3 broker Kafka cluster and when we're trying to collect the
> > > metrics like messages in per sec, bytes in per sec, etc. we get the
> > values
> > > as 0 for one of the three brokers. But we get proper values for metrics
> > > like heap memory usage for all the brokers. When we restart the cluster,
> > > the same or some other broker would behave in a similar way.
> > >
> > > We're seeing similar behavior in another cluster as well.
> > >
> > > 2) We're logging the time it takes to collect the metrics. The time to
> > > collect seems to increase over time and crosses a minute in a couple of
> > > days. It's of the order of 1 or 2 seconds when start the cluster.
> > >
> > > Thanks.
> > >
> >