You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kyle Banker <ky...@gmail.com> on 2015/01/26 23:42:40 UTC
Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
I've been using a custom KafkaMetricsReporter to report Kafka broker
metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages in
and out for all topics together and for each individual topic.
After upgrading to v0.8.2.0, these metrics are no longer being reported.
I'm only seeing the following:
BrokerTopicMetrics
- BytesInPerSec
- BytesOutPerSec
- BytesRejectedPerSec
- MessagesInPerSec
What's more, despite lots of successful writes to the cluster, the values
for these remaining metrics are all zero.
I saw that there was some refactoring of metric naming code. Was the
behavior supposed to have changed?
Many thanks in advance.
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Joel Koshy <jj...@gmail.com>.
Is it actually getting double-counted? I tried reproducing this
locally but the BrokerTopicMetrics.Count lines up with the sum of the
PerTopic.Counts for various metrics.
On Tue, Jan 27, 2015 at 03:29:37AM -0500, Jason Rosenberg wrote:
> Ok,
>
> It looks like the yammer MetricName is not being created correctly for the
> sub metrics that include a topic. E.g. a metric with an mbeanName like:
>
> kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
>
> appears to be malformed. A yammer MetricName has 4 fields that are used in
> creating a graphite metric, that are included in the constructor:
> group, type, name, scope.
>
> In this case, the metric with the above mbeanName has these fields set in
> the MetricName:
>
> group: "kafka.server"
> type: "BrokerTopicMetrics"
> name: "BytesInPerSec"
> scope: null
>
> Thus, the topic metrics all look the same, and get lumped into the
> top-level BrokerTopicMetrics (and thus that will now be double counted). It
> looks like the fix for kafka-1481 was where things got broken. It seems to
> have introduced ‘tags’ in the building of metric names, and then those tags
> only get applied to the mbeanName, but get excluded from the metric name:
> https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
>
> This is a pretty severe issue, since the yammer metrics for these stats
> will be double counted in aggregate, and the per-topic stats will be
> removed.
>
> I should note too, in my previous email, I thought that only the per-topic
> BrokerTopicMetrics were missing, but also several other per-topic metrics
> are missing too, e.g. under kafka.log, etc.
>
> Jason
>
>
> On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <jb...@squareup.com> wrote:
>
> > I can confirm that the per topic metrics are not coming through to the
> > yammer metrics registry. I do see them in jmx (via jconsole), but the
> > MetricsRegistry does not have them.
> > All the other metrics are coming through that appear in jmx.
> >
> > This is with single node instance running locally.
> >
> > Jason
> >
> >
> >
> > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <ku...@nmsworks.co.in>
> > wrote:
> >
> >> If you are using multi-node cluster, then metrics may be reported from
> >> other servers.
> >> pl check all the servers in the cluster.
> >>
> >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <ky...@gmail.com>
> >> wrote:
> >>
> >> > I've been using a custom KafkaMetricsReporter to report Kafka broker
> >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
> >> messages in
> >> > and out for all topics together and for each individual topic.
> >> >
> >> > After upgrading to v0.8.2.0, these metrics are no longer being reported.
> >> >
> >> > I'm only seeing the following:
> >> > BrokerTopicMetrics
> >> > - BytesInPerSec
> >> > - BytesOutPerSec
> >> > - BytesRejectedPerSec
> >> > - MessagesInPerSec
> >> >
> >> > What's more, despite lots of successful writes to the cluster, the
> >> values
> >> > for these remaining metrics are all zero.
> >> >
> >> > I saw that there was some refactoring of metric naming code. Was the
> >> > behavior supposed to have changed?
> >> >
> >> > Many thanks in advance.
> >> >
> >>
> >
> >
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Kyle Banker <ky...@gmail.com>.
Thanks for the quick response, Jun (and many thanks to Jason for confirming
and further investigating the issue). I've tested the patch, and it does
fix the fundamental issue, but I do have a few comments that I'll leave on
the ticket.
On Tue, Jan 27, 2015 at 9:19 AM, Jun Rao <ju...@confluent.io> wrote:
> Jason, Kyle,
>
> I created an 0.8.2 blocker
> https://issues.apache.org/jira/browse/KAFKA-1902
> and attached a patch there. Could you test it out and see if it fixes the
> issue with the reporter? The patch adds tags as scope in MetricName.
>
> Thanks,
>
> Jun
>
> On Tue, Jan 27, 2015 at 7:39 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Jason,
> >
> > So, this sounds like a real issue. Perhaps we can fix it just by setting
> > the tag name as the scope. For example, for mbean kafka.server:type=
> > BrokerTopicMetrics,name=BytesInPerSec,topic=test, we can have
> >
> > group: "kafka.server"
> > type: "BrokerTopicMetrics"
> > name: "BytesInPerSec"
> > scope: "topic=test"
> >
> > Do you know if scope can have characters like "=" and "," (e.g., for
> scope
> > like "topic=test,partition=1")?
> >
> > The issue with using mytopic-BytesInPerSec as the name is what we are
> > trying to fix in kafka-1481. Topic name (and clientId, etc) can have dash
> > in it and it's hard to parse.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Tue, Jan 27, 2015 at 6:30 AM, Jason Rosenberg <jb...@squareup.com>
> wrote:
> >
> >> Remember multiple people have reported this issue. Per topic metrics no
> >> longer appear in graphite (or in any system modeled after the yammer
> >> GraphiteReporter). They are not being seen as unique.
> >>
> >> While these metrics are registered in the registry as separate
> >> ‘MetricName’
> >> instances (varying only by mbeanName), the GraphiteReporter sends the
> >> metrics to graphite using only the 4 fields I describe above.
> >> Consequently,
> >> multiple metrics in the registry get sent to graphite under the same
> name.
> >> Thus these metrics all end up in the same bucket in graphite, trampling
> >> over each other making them useless. They aren’t ‘double counted’ so
> much
> >> as flapping between multiple independent values.
> >>
> >> We actually have our own Reporter class (based off the yammer
> >> GraphiteReporter). Our version sends metrics through kafka which is then
> >> consumed downstream by multiple metric consumers.
> >>
> >> The ConsoleReporter isn’t useful for actually persisting metrics
> anywhere.
> >> It’s just iterating through all the (identically named metrics in the
> >> registry (save for the different mbeanNames))….
> >>
> >> The mbeanName, as constructed, is not useful as a human readable metric
> >> name, to be presented in a browsable tree of metrics, etc. The
> >> ‘group’:’type’:’name’:’scope’ are the pieces that matter.
> >>
> >> The fix here is to produce MetricName instances similar to 0.8.1.1, etc.
> >> In
> >> this case, it should probably be something like:
> >>
> >> group: kafka.server
> >> type: BrokerTopicMetrics
> >> name: mytopic-BytesInPerSec
> >> group: <unused>
> >>
> >> Jason
> >>
> >>
> >> On Tue, Jan 27, 2015 at 7:26 AM, Manikumar Reddy <ku...@nmsworks.co.in>
> >> wrote:
> >>
> >> > I have enabled yammer's ConsoleReporter and I am getting all the
> metrics
> >> > (including per-topic metrics).
> >> >
> >> > Yammer's MetricName object implements equals/hashcode methods using
> >> > mBeanName . We are constructing a unique mBeanName for each metric, So
> >> we
> >> > are not missing/overwriting any metrics.
> >> >
> >> > Current confusion is due to MetricName.name(). This will be same
> >> > (BytesInPerSec) for both broker level and topic level metrics. We need
> >> to
> >> > use MetricName.getMBeanName() to differentiate between broker level
> and
> >> > topic level metrics.
> >> >
> >> > 0.8.1 MBeanName:
> >> > "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec"
> >> > "kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec"
> >> >
> >> > 0.8.2 MBeanName:
> >> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
> >> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC
> >> >
> >> >
> >> > ConsoleReporter's O/P:
> >> >
> >> > BytesInPerSec: <- This is broker level
> >> > count = 1521
> >> > mean rate = 3.63 bytes/s
> >> > 1-minute rate = 0.35 bytes/s
> >> > 5-minute rate = 2.07 bytes/s
> >> > 15-minute rate = 1.25 bytes/s
> >> >
> >> > BytesInPerSec: <- This is for topic1
> >> > count = 626
> >> > mean rate = 1.89 bytes/s
> >> > 1-minute rate = 0.42 bytes/s
> >> > 5-minute rate = 31.53 bytes/s
> >> > 15-minute rate = 64.66 bytes/s
> >> >
> >> > BytesInPerSec: <- This is for topic2
> >> > count = 895
> >> > mean rate = 3.62 bytes/s
> >> > 1-minute rate = 1.39 bytes/s
> >> > 5-minute rate = 30.08 bytes/s
> >> > 15-minute rate = 50.27 bytes/s
> >> >
> >> > Manikumar
> >> >
> >> > On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <jb...@squareup.com>
> >> wrote:
> >> >
> >> > > Ok,
> >> > >
> >> > > It looks like the yammer MetricName is not being created correctly
> for
> >> > the
> >> > > sub metrics that include a topic. E.g. a metric with an mbeanName
> >> like:
> >> > >
> >> > >
> >> >
> >>
> kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
> >> > >
> >> > > appears to be malformed. A yammer MetricName has 4 fields that are
> >> used
> >> > in
> >> > > creating a graphite metric, that are included in the constructor:
> >> > > group, type, name, scope.
> >> > >
> >> > > In this case, the metric with the above mbeanName has these fields
> >> set in
> >> > > the MetricName:
> >> > >
> >> > > group: "kafka.server"
> >> > > type: "BrokerTopicMetrics"
> >> > > name: "BytesInPerSec"
> >> > > scope: null
> >> > >
> >> > > Thus, the topic metrics all look the same, and get lumped into the
> >> > > top-level BrokerTopicMetrics (and thus that will now be double
> >> counted).
> >> > It
> >> > > looks like the fix for kafka-1481 was where things got broken. It
> >> seems
> >> > to
> >> > > have introduced ‘tags’ in the building of metric names, and then
> those
> >> > tags
> >> > > only get applied to the mbeanName, but get excluded from the metric
> >> name:
> >> > >
> >> > >
> >> >
> >>
> https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
> >> > >
> >> > > This is a pretty severe issue, since the yammer metrics for these
> >> stats
> >> > > will be double counted in aggregate, and the per-topic stats will be
> >> > > removed.
> >> > >
> >> > > I should note too, in my previous email, I thought that only the
> >> > per-topic
> >> > > BrokerTopicMetrics were missing, but also several other per-topic
> >> metrics
> >> > > are missing too, e.g. under kafka.log, etc.
> >> > >
> >> > > Jason
> >> > >
> >> > >
> >> > > On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <jb...@squareup.com>
> >> > wrote:
> >> > >
> >> > > > I can confirm that the per topic metrics are not coming through to
> >> the
> >> > > > yammer metrics registry. I do see them in jmx (via jconsole), but
> >> the
> >> > > > MetricsRegistry does not have them.
> >> > > > All the other metrics are coming through that appear in jmx.
> >> > > >
> >> > > > This is with single node instance running locally.
> >> > > >
> >> > > > Jason
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <
> >> kumar@nmsworks.co.in
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > >> If you are using multi-node cluster, then metrics may be reported
> >> from
> >> > > >> other servers.
> >> > > >> pl check all the servers in the cluster.
> >> > > >>
> >> > > >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <
> kylebanker@gmail.com
> >> >
> >> > > >> wrote:
> >> > > >>
> >> > > >> > I've been using a custom KafkaMetricsReporter to report Kafka
> >> broker
> >> > > >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
> >> > > >> messages in
> >> > > >> > and out for all topics together and for each individual topic.
> >> > > >> >
> >> > > >> > After upgrading to v0.8.2.0, these metrics are no longer being
> >> > > reported.
> >> > > >> >
> >> > > >> > I'm only seeing the following:
> >> > > >> > BrokerTopicMetrics
> >> > > >> > - BytesInPerSec
> >> > > >> > - BytesOutPerSec
> >> > > >> > - BytesRejectedPerSec
> >> > > >> > - MessagesInPerSec
> >> > > >> >
> >> > > >> > What's more, despite lots of successful writes to the cluster,
> >> the
> >> > > >> values
> >> > > >> > for these remaining metrics are all zero.
> >> > > >> >
> >> > > >> > I saw that there was some refactoring of metric naming code.
> Was
> >> the
> >> > > >> > behavior supposed to have changed?
> >> > > >> >
> >> > > >> > Many thanks in advance.
> >> > > >> >
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Jun Rao <ju...@confluent.io>.
Jason, Kyle,
Added comments to the jira. Let me know if they make sense. The dot
convention is a bit tricky to follow since we allow dots in topic and
clientId, etc. Also, we probably don't want to use a convention too
specific for Graphite since other systems may have other conventions.
Thanks,
Jun
On Tue, Jan 27, 2015 at 9:32 AM, Jason Rosenberg <jb...@squareup.com> wrote:
> I added a comment to the ticket. I think it will work getting data
> disambiguated (didn't actually test end to end to graphite).
> However, the naming scheme is not ideal for how metric ui's typically would
> present the metric tree (e.g. jmx tag syntax doesn't really translate).
>
> Jason
>
> On Tue, Jan 27, 2015 at 11:19 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Jason, Kyle,
> >
> > I created an 0.8.2 blocker
> > https://issues.apache.org/jira/browse/KAFKA-1902
> > and attached a patch there. Could you test it out and see if it fixes the
> > issue with the reporter? The patch adds tags as scope in MetricName.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Jan 27, 2015 at 7:39 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Jason,
> > >
> > > So, this sounds like a real issue. Perhaps we can fix it just by
> setting
> > > the tag name as the scope. For example, for mbean kafka.server:type=
> > > BrokerTopicMetrics,name=BytesInPerSec,topic=test, we can have
> > >
> > > group: "kafka.server"
> > > type: "BrokerTopicMetrics"
> > > name: "BytesInPerSec"
> > > scope: "topic=test"
> > >
> > > Do you know if scope can have characters like "=" and "," (e.g., for
> > scope
> > > like "topic=test,partition=1")?
> > >
> > > The issue with using mytopic-BytesInPerSec as the name is what we are
> > > trying to fix in kafka-1481. Topic name (and clientId, etc) can have
> dash
> > > in it and it's hard to parse.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Tue, Jan 27, 2015 at 6:30 AM, Jason Rosenberg <jb...@squareup.com>
> > wrote:
> > >
> > >> Remember multiple people have reported this issue. Per topic metrics
> no
> > >> longer appear in graphite (or in any system modeled after the yammer
> > >> GraphiteReporter). They are not being seen as unique.
> > >>
> > >> While these metrics are registered in the registry as separate
> > >> ‘MetricName’
> > >> instances (varying only by mbeanName), the GraphiteReporter sends the
> > >> metrics to graphite using only the 4 fields I describe above.
> > >> Consequently,
> > >> multiple metrics in the registry get sent to graphite under the same
> > name.
> > >> Thus these metrics all end up in the same bucket in graphite,
> trampling
> > >> over each other making them useless. They aren’t ‘double counted’ so
> > much
> > >> as flapping between multiple independent values.
> > >>
> > >> We actually have our own Reporter class (based off the yammer
> > >> GraphiteReporter). Our version sends metrics through kafka which is
> then
> > >> consumed downstream by multiple metric consumers.
> > >>
> > >> The ConsoleReporter isn’t useful for actually persisting metrics
> > anywhere.
> > >> It’s just iterating through all the (identically named metrics in the
> > >> registry (save for the different mbeanNames))….
> > >>
> > >> The mbeanName, as constructed, is not useful as a human readable
> metric
> > >> name, to be presented in a browsable tree of metrics, etc. The
> > >> ‘group’:’type’:’name’:’scope’ are the pieces that matter.
> > >>
> > >> The fix here is to produce MetricName instances similar to 0.8.1.1,
> etc.
> > >> In
> > >> this case, it should probably be something like:
> > >>
> > >> group: kafka.server
> > >> type: BrokerTopicMetrics
> > >> name: mytopic-BytesInPerSec
> > >> group: <unused>
> > >>
> > >> Jason
> > >>
> > >>
> > >> On Tue, Jan 27, 2015 at 7:26 AM, Manikumar Reddy <
> kumar@nmsworks.co.in>
> > >> wrote:
> > >>
> > >> > I have enabled yammer's ConsoleReporter and I am getting all the
> > metrics
> > >> > (including per-topic metrics).
> > >> >
> > >> > Yammer's MetricName object implements equals/hashcode methods using
> > >> > mBeanName . We are constructing a unique mBeanName for each metric,
> So
> > >> we
> > >> > are not missing/overwriting any metrics.
> > >> >
> > >> > Current confusion is due to MetricName.name(). This will be same
> > >> > (BytesInPerSec) for both broker level and topic level metrics. We
> need
> > >> to
> > >> > use MetricName.getMBeanName() to differentiate between broker level
> > and
> > >> > topic level metrics.
> > >> >
> > >> > 0.8.1 MBeanName:
> > >> >
> "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec"
> > >> >
> "kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec"
> > >> >
> > >> > 0.8.2 MBeanName:
> > >> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
> > >> >
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC
> > >> >
> > >> >
> > >> > ConsoleReporter's O/P:
> > >> >
> > >> > BytesInPerSec: <- This is broker level
> > >> > count = 1521
> > >> > mean rate = 3.63 bytes/s
> > >> > 1-minute rate = 0.35 bytes/s
> > >> > 5-minute rate = 2.07 bytes/s
> > >> > 15-minute rate = 1.25 bytes/s
> > >> >
> > >> > BytesInPerSec: <- This is for topic1
> > >> > count = 626
> > >> > mean rate = 1.89 bytes/s
> > >> > 1-minute rate = 0.42 bytes/s
> > >> > 5-minute rate = 31.53 bytes/s
> > >> > 15-minute rate = 64.66 bytes/s
> > >> >
> > >> > BytesInPerSec: <- This is for topic2
> > >> > count = 895
> > >> > mean rate = 3.62 bytes/s
> > >> > 1-minute rate = 1.39 bytes/s
> > >> > 5-minute rate = 30.08 bytes/s
> > >> > 15-minute rate = 50.27 bytes/s
> > >> >
> > >> > Manikumar
> > >> >
> > >> > On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <jb...@squareup.com>
> > >> wrote:
> > >> >
> > >> > > Ok,
> > >> > >
> > >> > > It looks like the yammer MetricName is not being created correctly
> > for
> > >> > the
> > >> > > sub metrics that include a topic. E.g. a metric with an mbeanName
> > >> like:
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
> > >> > >
> > >> > > appears to be malformed. A yammer MetricName has 4 fields that are
> > >> used
> > >> > in
> > >> > > creating a graphite metric, that are included in the constructor:
> > >> > > group, type, name, scope.
> > >> > >
> > >> > > In this case, the metric with the above mbeanName has these fields
> > >> set in
> > >> > > the MetricName:
> > >> > >
> > >> > > group: "kafka.server"
> > >> > > type: "BrokerTopicMetrics"
> > >> > > name: "BytesInPerSec"
> > >> > > scope: null
> > >> > >
> > >> > > Thus, the topic metrics all look the same, and get lumped into the
> > >> > > top-level BrokerTopicMetrics (and thus that will now be double
> > >> counted).
> > >> > It
> > >> > > looks like the fix for kafka-1481 was where things got broken. It
> > >> seems
> > >> > to
> > >> > > have introduced ‘tags’ in the building of metric names, and then
> > those
> > >> > tags
> > >> > > only get applied to the mbeanName, but get excluded from the
> metric
> > >> name:
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
> > >> > >
> > >> > > This is a pretty severe issue, since the yammer metrics for these
> > >> stats
> > >> > > will be double counted in aggregate, and the per-topic stats will
> be
> > >> > > removed.
> > >> > >
> > >> > > I should note too, in my previous email, I thought that only the
> > >> > per-topic
> > >> > > BrokerTopicMetrics were missing, but also several other per-topic
> > >> metrics
> > >> > > are missing too, e.g. under kafka.log, etc.
> > >> > >
> > >> > > Jason
> > >> > >
> > >> > >
> > >> > > On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <
> jbr@squareup.com>
> > >> > wrote:
> > >> > >
> > >> > > > I can confirm that the per topic metrics are not coming through
> to
> > >> the
> > >> > > > yammer metrics registry. I do see them in jmx (via jconsole),
> but
> > >> the
> > >> > > > MetricsRegistry does not have them.
> > >> > > > All the other metrics are coming through that appear in jmx.
> > >> > > >
> > >> > > > This is with single node instance running locally.
> > >> > > >
> > >> > > > Jason
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <
> > >> kumar@nmsworks.co.in
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > >> If you are using multi-node cluster, then metrics may be
> reported
> > >> from
> > >> > > >> other servers.
> > >> > > >> pl check all the servers in the cluster.
> > >> > > >>
> > >> > > >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <
> > kylebanker@gmail.com
> > >> >
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> > I've been using a custom KafkaMetricsReporter to report Kafka
> > >> broker
> > >> > > >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes
> and
> > >> > > >> messages in
> > >> > > >> > and out for all topics together and for each individual
> topic.
> > >> > > >> >
> > >> > > >> > After upgrading to v0.8.2.0, these metrics are no longer
> being
> > >> > > reported.
> > >> > > >> >
> > >> > > >> > I'm only seeing the following:
> > >> > > >> > BrokerTopicMetrics
> > >> > > >> > - BytesInPerSec
> > >> > > >> > - BytesOutPerSec
> > >> > > >> > - BytesRejectedPerSec
> > >> > > >> > - MessagesInPerSec
> > >> > > >> >
> > >> > > >> > What's more, despite lots of successful writes to the
> cluster,
> > >> the
> > >> > > >> values
> > >> > > >> > for these remaining metrics are all zero.
> > >> > > >> >
> > >> > > >> > I saw that there was some refactoring of metric naming code.
> > Was
> > >> the
> > >> > > >> > behavior supposed to have changed?
> > >> > > >> >
> > >> > > >> > Many thanks in advance.
> > >> > > >> >
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Jason Rosenberg <jb...@squareup.com>.
I added a comment to the ticket. I think it will work getting data
disambiguated (didn't actually test end to end to graphite).
However, the naming scheme is not ideal for how metric ui's typically would
present the metric tree (e.g. jmx tag syntax doesn't really translate).
Jason
On Tue, Jan 27, 2015 at 11:19 AM, Jun Rao <ju...@confluent.io> wrote:
> Jason, Kyle,
>
> I created an 0.8.2 blocker
> https://issues.apache.org/jira/browse/KAFKA-1902
> and attached a patch there. Could you test it out and see if it fixes the
> issue with the reporter? The patch adds tags as scope in MetricName.
>
> Thanks,
>
> Jun
>
> On Tue, Jan 27, 2015 at 7:39 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Jason,
> >
> > So, this sounds like a real issue. Perhaps we can fix it just by setting
> > the tag name as the scope. For example, for mbean kafka.server:type=
> > BrokerTopicMetrics,name=BytesInPerSec,topic=test, we can have
> >
> > group: "kafka.server"
> > type: "BrokerTopicMetrics"
> > name: "BytesInPerSec"
> > scope: "topic=test"
> >
> > Do you know if scope can have characters like "=" and "," (e.g., for
> scope
> > like "topic=test,partition=1")?
> >
> > The issue with using mytopic-BytesInPerSec as the name is what we are
> > trying to fix in kafka-1481. Topic name (and clientId, etc) can have dash
> > in it and it's hard to parse.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Tue, Jan 27, 2015 at 6:30 AM, Jason Rosenberg <jb...@squareup.com>
> wrote:
> >
> >> Remember multiple people have reported this issue. Per topic metrics no
> >> longer appear in graphite (or in any system modeled after the yammer
> >> GraphiteReporter). They are not being seen as unique.
> >>
> >> While these metrics are registered in the registry as separate
> >> ‘MetricName’
> >> instances (varying only by mbeanName), the GraphiteReporter sends the
> >> metrics to graphite using only the 4 fields I describe above.
> >> Consequently,
> >> multiple metrics in the registry get sent to graphite under the same
> name.
> >> Thus these metrics all end up in the same bucket in graphite, trampling
> >> over each other making them useless. They aren’t ‘double counted’ so
> much
> >> as flapping between multiple independent values.
> >>
> >> We actually have our own Reporter class (based off the yammer
> >> GraphiteReporter). Our version sends metrics through kafka which is then
> >> consumed downstream by multiple metric consumers.
> >>
> >> The ConsoleReporter isn’t useful for actually persisting metrics
> anywhere.
> >> It’s just iterating through all the (identically named metrics in the
> >> registry (save for the different mbeanNames))….
> >>
> >> The mbeanName, as constructed, is not useful as a human readable metric
> >> name, to be presented in a browsable tree of metrics, etc. The
> >> ‘group’:’type’:’name’:’scope’ are the pieces that matter.
> >>
> >> The fix here is to produce MetricName instances similar to 0.8.1.1, etc.
> >> In
> >> this case, it should probably be something like:
> >>
> >> group: kafka.server
> >> type: BrokerTopicMetrics
> >> name: mytopic-BytesInPerSec
> >> group: <unused>
> >>
> >> Jason
> >>
> >>
> >> On Tue, Jan 27, 2015 at 7:26 AM, Manikumar Reddy <ku...@nmsworks.co.in>
> >> wrote:
> >>
> >> > I have enabled yammer's ConsoleReporter and I am getting all the
> metrics
> >> > (including per-topic metrics).
> >> >
> >> > Yammer's MetricName object implements equals/hashcode methods using
> >> > mBeanName . We are constructing a unique mBeanName for each metric, So
> >> we
> >> > are not missing/overwriting any metrics.
> >> >
> >> > Current confusion is due to MetricName.name(). This will be same
> >> > (BytesInPerSec) for both broker level and topic level metrics. We need
> >> to
> >> > use MetricName.getMBeanName() to differentiate between broker level
> and
> >> > topic level metrics.
> >> >
> >> > 0.8.1 MBeanName:
> >> > "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec"
> >> > "kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec"
> >> >
> >> > 0.8.2 MBeanName:
> >> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
> >> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC
> >> >
> >> >
> >> > ConsoleReporter's O/P:
> >> >
> >> > BytesInPerSec: <- This is broker level
> >> > count = 1521
> >> > mean rate = 3.63 bytes/s
> >> > 1-minute rate = 0.35 bytes/s
> >> > 5-minute rate = 2.07 bytes/s
> >> > 15-minute rate = 1.25 bytes/s
> >> >
> >> > BytesInPerSec: <- This is for topic1
> >> > count = 626
> >> > mean rate = 1.89 bytes/s
> >> > 1-minute rate = 0.42 bytes/s
> >> > 5-minute rate = 31.53 bytes/s
> >> > 15-minute rate = 64.66 bytes/s
> >> >
> >> > BytesInPerSec: <- This is for topic2
> >> > count = 895
> >> > mean rate = 3.62 bytes/s
> >> > 1-minute rate = 1.39 bytes/s
> >> > 5-minute rate = 30.08 bytes/s
> >> > 15-minute rate = 50.27 bytes/s
> >> >
> >> > Manikumar
> >> >
> >> > On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <jb...@squareup.com>
> >> wrote:
> >> >
> >> > > Ok,
> >> > >
> >> > > It looks like the yammer MetricName is not being created correctly
> for
> >> > the
> >> > > sub metrics that include a topic. E.g. a metric with an mbeanName
> >> like:
> >> > >
> >> > >
> >> >
> >>
> kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
> >> > >
> >> > > appears to be malformed. A yammer MetricName has 4 fields that are
> >> used
> >> > in
> >> > > creating a graphite metric, that are included in the constructor:
> >> > > group, type, name, scope.
> >> > >
> >> > > In this case, the metric with the above mbeanName has these fields
> >> set in
> >> > > the MetricName:
> >> > >
> >> > > group: "kafka.server"
> >> > > type: "BrokerTopicMetrics"
> >> > > name: "BytesInPerSec"
> >> > > scope: null
> >> > >
> >> > > Thus, the topic metrics all look the same, and get lumped into the
> >> > > top-level BrokerTopicMetrics (and thus that will now be double
> >> counted).
> >> > It
> >> > > looks like the fix for kafka-1481 was where things got broken. It
> >> seems
> >> > to
> >> > > have introduced ‘tags’ in the building of metric names, and then
> those
> >> > tags
> >> > > only get applied to the mbeanName, but get excluded from the metric
> >> name:
> >> > >
> >> > >
> >> >
> >>
> https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
> >> > >
> >> > > This is a pretty severe issue, since the yammer metrics for these
> >> stats
> >> > > will be double counted in aggregate, and the per-topic stats will be
> >> > > removed.
> >> > >
> >> > > I should note too, in my previous email, I thought that only the
> >> > per-topic
> >> > > BrokerTopicMetrics were missing, but also several other per-topic
> >> metrics
> >> > > are missing too, e.g. under kafka.log, etc.
> >> > >
> >> > > Jason
> >> > >
> >> > >
> >> > > On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <jb...@squareup.com>
> >> > wrote:
> >> > >
> >> > > > I can confirm that the per topic metrics are not coming through to
> >> the
> >> > > > yammer metrics registry. I do see them in jmx (via jconsole), but
> >> the
> >> > > > MetricsRegistry does not have them.
> >> > > > All the other metrics are coming through that appear in jmx.
> >> > > >
> >> > > > This is with single node instance running locally.
> >> > > >
> >> > > > Jason
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <
> >> kumar@nmsworks.co.in
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > >> If you are using multi-node cluster, then metrics may be reported
> >> from
> >> > > >> other servers.
> >> > > >> pl check all the servers in the cluster.
> >> > > >>
> >> > > >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <
> kylebanker@gmail.com
> >> >
> >> > > >> wrote:
> >> > > >>
> >> > > >> > I've been using a custom KafkaMetricsReporter to report Kafka
> >> broker
> >> > > >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
> >> > > >> messages in
> >> > > >> > and out for all topics together and for each individual topic.
> >> > > >> >
> >> > > >> > After upgrading to v0.8.2.0, these metrics are no longer being
> >> > > reported.
> >> > > >> >
> >> > > >> > I'm only seeing the following:
> >> > > >> > BrokerTopicMetrics
> >> > > >> > - BytesInPerSec
> >> > > >> > - BytesOutPerSec
> >> > > >> > - BytesRejectedPerSec
> >> > > >> > - MessagesInPerSec
> >> > > >> >
> >> > > >> > What's more, despite lots of successful writes to the cluster,
> >> the
> >> > > >> values
> >> > > >> > for these remaining metrics are all zero.
> >> > > >> >
> >> > > >> > I saw that there was some refactoring of metric naming code.
> Was
> >> the
> >> > > >> > behavior supposed to have changed?
> >> > > >> >
> >> > > >> > Many thanks in advance.
> >> > > >> >
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Jun Rao <ju...@confluent.io>.
Jason, Kyle,
I created an 0.8.2 blocker https://issues.apache.org/jira/browse/KAFKA-1902
and attached a patch there. Could you test it out and see if it fixes the
issue with the reporter? The patch adds tags as scope in MetricName.
Thanks,
Jun
On Tue, Jan 27, 2015 at 7:39 AM, Jun Rao <ju...@confluent.io> wrote:
> Jason,
>
> So, this sounds like a real issue. Perhaps we can fix it just by setting
> the tag name as the scope. For example, for mbean kafka.server:type=
> BrokerTopicMetrics,name=BytesInPerSec,topic=test, we can have
>
> group: "kafka.server"
> type: "BrokerTopicMetrics"
> name: "BytesInPerSec"
> scope: "topic=test"
>
> Do you know if scope can have characters like "=" and "," (e.g., for scope
> like "topic=test,partition=1")?
>
> The issue with using mytopic-BytesInPerSec as the name is what we are
> trying to fix in kafka-1481. Topic name (and clientId, etc) can have dash
> in it and it's hard to parse.
>
> Thanks,
>
> Jun
>
>
>
> On Tue, Jan 27, 2015 at 6:30 AM, Jason Rosenberg <jb...@squareup.com> wrote:
>
>> Remember multiple people have reported this issue. Per topic metrics no
>> longer appear in graphite (or in any system modeled after the yammer
>> GraphiteReporter). They are not being seen as unique.
>>
>> While these metrics are registered in the registry as separate
>> ‘MetricName’
>> instances (varying only by mbeanName), the GraphiteReporter sends the
>> metrics to graphite using only the 4 fields I describe above.
>> Consequently,
>> multiple metrics in the registry get sent to graphite under the same name.
>> Thus these metrics all end up in the same bucket in graphite, trampling
>> over each other making them useless. They aren’t ‘double counted’ so much
>> as flapping between multiple independent values.
>>
>> We actually have our own Reporter class (based off the yammer
>> GraphiteReporter). Our version sends metrics through kafka which is then
>> consumed downstream by multiple metric consumers.
>>
>> The ConsoleReporter isn’t useful for actually persisting metrics anywhere.
>> It’s just iterating through all the (identically named metrics in the
>> registry (save for the different mbeanNames))….
>>
>> The mbeanName, as constructed, is not useful as a human readable metric
>> name, to be presented in a browsable tree of metrics, etc. The
>> ‘group’:’type’:’name’:’scope’ are the pieces that matter.
>>
>> The fix here is to produce MetricName instances similar to 0.8.1.1, etc.
>> In
>> this case, it should probably be something like:
>>
>> group: kafka.server
>> type: BrokerTopicMetrics
>> name: mytopic-BytesInPerSec
>> group: <unused>
>>
>> Jason
>>
>>
>> On Tue, Jan 27, 2015 at 7:26 AM, Manikumar Reddy <ku...@nmsworks.co.in>
>> wrote:
>>
>> > I have enabled yammer's ConsoleReporter and I am getting all the metrics
>> > (including per-topic metrics).
>> >
>> > Yammer's MetricName object implements equals/hashcode methods using
>> > mBeanName . We are constructing a unique mBeanName for each metric, So
>> we
>> > are not missing/overwriting any metrics.
>> >
>> > Current confusion is due to MetricName.name(). This will be same
>> > (BytesInPerSec) for both broker level and topic level metrics. We need
>> to
>> > use MetricName.getMBeanName() to differentiate between broker level and
>> > topic level metrics.
>> >
>> > 0.8.1 MBeanName:
>> > "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec"
>> > "kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec"
>> >
>> > 0.8.2 MBeanName:
>> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
>> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC
>> >
>> >
>> > ConsoleReporter's O/P:
>> >
>> > BytesInPerSec: <- This is broker level
>> > count = 1521
>> > mean rate = 3.63 bytes/s
>> > 1-minute rate = 0.35 bytes/s
>> > 5-minute rate = 2.07 bytes/s
>> > 15-minute rate = 1.25 bytes/s
>> >
>> > BytesInPerSec: <- This is for topic1
>> > count = 626
>> > mean rate = 1.89 bytes/s
>> > 1-minute rate = 0.42 bytes/s
>> > 5-minute rate = 31.53 bytes/s
>> > 15-minute rate = 64.66 bytes/s
>> >
>> > BytesInPerSec: <- This is for topic2
>> > count = 895
>> > mean rate = 3.62 bytes/s
>> > 1-minute rate = 1.39 bytes/s
>> > 5-minute rate = 30.08 bytes/s
>> > 15-minute rate = 50.27 bytes/s
>> >
>> > Manikumar
>> >
>> > On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <jb...@squareup.com>
>> wrote:
>> >
>> > > Ok,
>> > >
>> > > It looks like the yammer MetricName is not being created correctly for
>> > the
>> > > sub metrics that include a topic. E.g. a metric with an mbeanName
>> like:
>> > >
>> > >
>> >
>> kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
>> > >
>> > > appears to be malformed. A yammer MetricName has 4 fields that are
>> used
>> > in
>> > > creating a graphite metric, that are included in the constructor:
>> > > group, type, name, scope.
>> > >
>> > > In this case, the metric with the above mbeanName has these fields
>> set in
>> > > the MetricName:
>> > >
>> > > group: "kafka.server"
>> > > type: "BrokerTopicMetrics"
>> > > name: "BytesInPerSec"
>> > > scope: null
>> > >
>> > > Thus, the topic metrics all look the same, and get lumped into the
>> > > top-level BrokerTopicMetrics (and thus that will now be double
>> counted).
>> > It
>> > > looks like the fix for kafka-1481 was where things got broken. It
>> seems
>> > to
>> > > have introduced ‘tags’ in the building of metric names, and then those
>> > tags
>> > > only get applied to the mbeanName, but get excluded from the metric
>> name:
>> > >
>> > >
>> >
>> https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
>> > >
>> > > This is a pretty severe issue, since the yammer metrics for these
>> stats
>> > > will be double counted in aggregate, and the per-topic stats will be
>> > > removed.
>> > >
>> > > I should note too, in my previous email, I thought that only the
>> > per-topic
>> > > BrokerTopicMetrics were missing, but also several other per-topic
>> metrics
>> > > are missing too, e.g. under kafka.log, etc.
>> > >
>> > > Jason
>> > >
>> > >
>> > > On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <jb...@squareup.com>
>> > wrote:
>> > >
>> > > > I can confirm that the per topic metrics are not coming through to
>> the
>> > > > yammer metrics registry. I do see them in jmx (via jconsole), but
>> the
>> > > > MetricsRegistry does not have them.
>> > > > All the other metrics are coming through that appear in jmx.
>> > > >
>> > > > This is with single node instance running locally.
>> > > >
>> > > > Jason
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <
>> kumar@nmsworks.co.in
>> > >
>> > > > wrote:
>> > > >
>> > > >> If you are using multi-node cluster, then metrics may be reported
>> from
>> > > >> other servers.
>> > > >> pl check all the servers in the cluster.
>> > > >>
>> > > >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <kylebanker@gmail.com
>> >
>> > > >> wrote:
>> > > >>
>> > > >> > I've been using a custom KafkaMetricsReporter to report Kafka
>> broker
>> > > >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
>> > > >> messages in
>> > > >> > and out for all topics together and for each individual topic.
>> > > >> >
>> > > >> > After upgrading to v0.8.2.0, these metrics are no longer being
>> > > reported.
>> > > >> >
>> > > >> > I'm only seeing the following:
>> > > >> > BrokerTopicMetrics
>> > > >> > - BytesInPerSec
>> > > >> > - BytesOutPerSec
>> > > >> > - BytesRejectedPerSec
>> > > >> > - MessagesInPerSec
>> > > >> >
>> > > >> > What's more, despite lots of successful writes to the cluster,
>> the
>> > > >> values
>> > > >> > for these remaining metrics are all zero.
>> > > >> >
>> > > >> > I saw that there was some refactoring of metric naming code. Was
>> the
>> > > >> > behavior supposed to have changed?
>> > > >> >
>> > > >> > Many thanks in advance.
>> > > >> >
>> > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Jun Rao <ju...@confluent.io>.
Jason,
So, this sounds like a real issue. Perhaps we can fix it just by setting
the tag name as the scope. For example, for mbean kafka.server:type=
BrokerTopicMetrics,name=BytesInPerSec,topic=test, we can have
group: "kafka.server"
type: "BrokerTopicMetrics"
name: "BytesInPerSec"
scope: "topic=test"
Do you know if scope can have characters like "=" and "," (e.g., for scope
like "topic=test,partition=1")?
The issue with using mytopic-BytesInPerSec as the name is what we are
trying to fix in kafka-1481. Topic name (and clientId, etc) can have dash
in it and it's hard to parse.
Thanks,
Jun
On Tue, Jan 27, 2015 at 6:30 AM, Jason Rosenberg <jb...@squareup.com> wrote:
> Remember multiple people have reported this issue. Per topic metrics no
> longer appear in graphite (or in any system modeled after the yammer
> GraphiteReporter). They are not being seen as unique.
>
> While these metrics are registered in the registry as separate ‘MetricName’
> instances (varying only by mbeanName), the GraphiteReporter sends the
> metrics to graphite using only the 4 fields I describe above. Consequently,
> multiple metrics in the registry get sent to graphite under the same name.
> Thus these metrics all end up in the same bucket in graphite, trampling
> over each other making them useless. They aren’t ‘double counted’ so much
> as flapping between multiple independent values.
>
> We actually have our own Reporter class (based off the yammer
> GraphiteReporter). Our version sends metrics through kafka which is then
> consumed downstream by multiple metric consumers.
>
> The ConsoleReporter isn’t useful for actually persisting metrics anywhere.
> It’s just iterating through all the (identically named metrics in the
> registry (save for the different mbeanNames))….
>
> The mbeanName, as constructed, is not useful as a human readable metric
> name, to be presented in a browsable tree of metrics, etc. The
> ‘group’:’type’:’name’:’scope’ are the pieces that matter.
>
> The fix here is to produce MetricName instances similar to 0.8.1.1, etc. In
> this case, it should probably be something like:
>
> group: kafka.server
> type: BrokerTopicMetrics
> name: mytopic-BytesInPerSec
> group: <unused>
>
> Jason
>
>
> On Tue, Jan 27, 2015 at 7:26 AM, Manikumar Reddy <ku...@nmsworks.co.in>
> wrote:
>
> > I have enabled yammer's ConsoleReporter and I am getting all the metrics
> > (including per-topic metrics).
> >
> > Yammer's MetricName object implements equals/hashcode methods using
> > mBeanName . We are constructing a unique mBeanName for each metric, So we
> > are not missing/overwriting any metrics.
> >
> > Current confusion is due to MetricName.name(). This will be same
> > (BytesInPerSec) for both broker level and topic level metrics. We need to
> > use MetricName.getMBeanName() to differentiate between broker level and
> > topic level metrics.
> >
> > 0.8.1 MBeanName:
> > "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec"
> > "kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec"
> >
> > 0.8.2 MBeanName:
> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC
> >
> >
> > ConsoleReporter's O/P:
> >
> > BytesInPerSec: <- This is broker level
> > count = 1521
> > mean rate = 3.63 bytes/s
> > 1-minute rate = 0.35 bytes/s
> > 5-minute rate = 2.07 bytes/s
> > 15-minute rate = 1.25 bytes/s
> >
> > BytesInPerSec: <- This is for topic1
> > count = 626
> > mean rate = 1.89 bytes/s
> > 1-minute rate = 0.42 bytes/s
> > 5-minute rate = 31.53 bytes/s
> > 15-minute rate = 64.66 bytes/s
> >
> > BytesInPerSec: <- This is for topic2
> > count = 895
> > mean rate = 3.62 bytes/s
> > 1-minute rate = 1.39 bytes/s
> > 5-minute rate = 30.08 bytes/s
> > 15-minute rate = 50.27 bytes/s
> >
> > Manikumar
> >
> > On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <jb...@squareup.com>
> wrote:
> >
> > > Ok,
> > >
> > > It looks like the yammer MetricName is not being created correctly for
> > the
> > > sub metrics that include a topic. E.g. a metric with an mbeanName like:
> > >
> > >
> >
> kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
> > >
> > > appears to be malformed. A yammer MetricName has 4 fields that are used
> > in
> > > creating a graphite metric, that are included in the constructor:
> > > group, type, name, scope.
> > >
> > > In this case, the metric with the above mbeanName has these fields set
> in
> > > the MetricName:
> > >
> > > group: "kafka.server"
> > > type: "BrokerTopicMetrics"
> > > name: "BytesInPerSec"
> > > scope: null
> > >
> > > Thus, the topic metrics all look the same, and get lumped into the
> > > top-level BrokerTopicMetrics (and thus that will now be double
> counted).
> > It
> > > looks like the fix for kafka-1481 was where things got broken. It seems
> > to
> > > have introduced ‘tags’ in the building of metric names, and then those
> > tags
> > > only get applied to the mbeanName, but get excluded from the metric
> name:
> > >
> > >
> >
> https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
> > >
> > > This is a pretty severe issue, since the yammer metrics for these stats
> > > will be double counted in aggregate, and the per-topic stats will be
> > > removed.
> > >
> > > I should note too, in my previous email, I thought that only the
> > per-topic
> > > BrokerTopicMetrics were missing, but also several other per-topic
> metrics
> > > are missing too, e.g. under kafka.log, etc.
> > >
> > > Jason
> > >
> > >
> > > On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <jb...@squareup.com>
> > wrote:
> > >
> > > > I can confirm that the per topic metrics are not coming through to
> the
> > > > yammer metrics registry. I do see them in jmx (via jconsole), but
> the
> > > > MetricsRegistry does not have them.
> > > > All the other metrics are coming through that appear in jmx.
> > > >
> > > > This is with single node instance running locally.
> > > >
> > > > Jason
> > > >
> > > >
> > > >
> > > > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <
> kumar@nmsworks.co.in
> > >
> > > > wrote:
> > > >
> > > >> If you are using multi-node cluster, then metrics may be reported
> from
> > > >> other servers.
> > > >> pl check all the servers in the cluster.
> > > >>
> > > >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <ky...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > I've been using a custom KafkaMetricsReporter to report Kafka
> broker
> > > >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
> > > >> messages in
> > > >> > and out for all topics together and for each individual topic.
> > > >> >
> > > >> > After upgrading to v0.8.2.0, these metrics are no longer being
> > > reported.
> > > >> >
> > > >> > I'm only seeing the following:
> > > >> > BrokerTopicMetrics
> > > >> > - BytesInPerSec
> > > >> > - BytesOutPerSec
> > > >> > - BytesRejectedPerSec
> > > >> > - MessagesInPerSec
> > > >> >
> > > >> > What's more, despite lots of successful writes to the cluster, the
> > > >> values
> > > >> > for these remaining metrics are all zero.
> > > >> >
> > > >> > I saw that there was some refactoring of metric naming code. Was
> the
> > > >> > behavior supposed to have changed?
> > > >> >
> > > >> > Many thanks in advance.
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Jason Rosenberg <jb...@squareup.com>.
Remember multiple people have reported this issue. Per topic metrics no
longer appear in graphite (or in any system modeled after the yammer
GraphiteReporter). They are not being seen as unique.
While these metrics are registered in the registry as separate ‘MetricName’
instances (varying only by mbeanName), the GraphiteReporter sends the
metrics to graphite using only the 4 fields I describe above. Consequently,
multiple metrics in the registry get sent to graphite under the same name.
Thus these metrics all end up in the same bucket in graphite, trampling
over each other making them useless. They aren’t ‘double counted’ so much
as flapping between multiple independent values.
We actually have our own Reporter class (based off the yammer
GraphiteReporter). Our version sends metrics through kafka which is then
consumed downstream by multiple metric consumers.
The ConsoleReporter isn’t useful for actually persisting metrics anywhere.
It’s just iterating through all the (identically named metrics in the
registry (save for the different mbeanNames))….
The mbeanName, as constructed, is not useful as a human readable metric
name, to be presented in a browsable tree of metrics, etc. The
‘group’:’type’:’name’:’scope’ are the pieces that matter.
The fix here is to produce MetricName instances similar to 0.8.1.1, etc. In
this case, it should probably be something like:
group: kafka.server
type: BrokerTopicMetrics
name: mytopic-BytesInPerSec
group: <unused>
Jason
On Tue, Jan 27, 2015 at 7:26 AM, Manikumar Reddy <ku...@nmsworks.co.in>
wrote:
> I have enabled yammer's ConsoleReporter and I am getting all the metrics
> (including per-topic metrics).
>
> Yammer's MetricName object implements equals/hashcode methods using
> mBeanName . We are constructing a unique mBeanName for each metric, So we
> are not missing/overwriting any metrics.
>
> Current confusion is due to MetricName.name(). This will be same
> (BytesInPerSec) for both broker level and topic level metrics. We need to
> use MetricName.getMBeanName() to differentiate between broker level and
> topic level metrics.
>
> 0.8.1 MBeanName:
> "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec"
> "kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec"
>
> 0.8.2 MBeanName:
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC
>
>
> ConsoleReporter's O/P:
>
> BytesInPerSec: <- This is broker level
> count = 1521
> mean rate = 3.63 bytes/s
> 1-minute rate = 0.35 bytes/s
> 5-minute rate = 2.07 bytes/s
> 15-minute rate = 1.25 bytes/s
>
> BytesInPerSec: <- This is for topic1
> count = 626
> mean rate = 1.89 bytes/s
> 1-minute rate = 0.42 bytes/s
> 5-minute rate = 31.53 bytes/s
> 15-minute rate = 64.66 bytes/s
>
> BytesInPerSec: <- This is for topic2
> count = 895
> mean rate = 3.62 bytes/s
> 1-minute rate = 1.39 bytes/s
> 5-minute rate = 30.08 bytes/s
> 15-minute rate = 50.27 bytes/s
>
> Manikumar
>
> On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <jb...@squareup.com> wrote:
>
> > Ok,
> >
> > It looks like the yammer MetricName is not being created correctly for
> the
> > sub metrics that include a topic. E.g. a metric with an mbeanName like:
> >
> >
> kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
> >
> > appears to be malformed. A yammer MetricName has 4 fields that are used
> in
> > creating a graphite metric, that are included in the constructor:
> > group, type, name, scope.
> >
> > In this case, the metric with the above mbeanName has these fields set in
> > the MetricName:
> >
> > group: "kafka.server"
> > type: "BrokerTopicMetrics"
> > name: "BytesInPerSec"
> > scope: null
> >
> > Thus, the topic metrics all look the same, and get lumped into the
> > top-level BrokerTopicMetrics (and thus that will now be double counted).
> It
> > looks like the fix for kafka-1481 was where things got broken. It seems
> to
> > have introduced ‘tags’ in the building of metric names, and then those
> tags
> > only get applied to the mbeanName, but get excluded from the metric name:
> >
> >
> https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
> >
> > This is a pretty severe issue, since the yammer metrics for these stats
> > will be double counted in aggregate, and the per-topic stats will be
> > removed.
> >
> > I should note too, in my previous email, I thought that only the
> per-topic
> > BrokerTopicMetrics were missing, but also several other per-topic metrics
> > are missing too, e.g. under kafka.log, etc.
> >
> > Jason
> >
> >
> > On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <jb...@squareup.com>
> wrote:
> >
> > > I can confirm that the per topic metrics are not coming through to the
> > > yammer metrics registry. I do see them in jmx (via jconsole), but the
> > > MetricsRegistry does not have them.
> > > All the other metrics are coming through that appear in jmx.
> > >
> > > This is with single node instance running locally.
> > >
> > > Jason
> > >
> > >
> > >
> > > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <kumar@nmsworks.co.in
> >
> > > wrote:
> > >
> > >> If you are using multi-node cluster, then metrics may be reported from
> > >> other servers.
> > >> pl check all the servers in the cluster.
> > >>
> > >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <ky...@gmail.com>
> > >> wrote:
> > >>
> > >> > I've been using a custom KafkaMetricsReporter to report Kafka broker
> > >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
> > >> messages in
> > >> > and out for all topics together and for each individual topic.
> > >> >
> > >> > After upgrading to v0.8.2.0, these metrics are no longer being
> > reported.
> > >> >
> > >> > I'm only seeing the following:
> > >> > BrokerTopicMetrics
> > >> > - BytesInPerSec
> > >> > - BytesOutPerSec
> > >> > - BytesRejectedPerSec
> > >> > - MessagesInPerSec
> > >> >
> > >> > What's more, despite lots of successful writes to the cluster, the
> > >> values
> > >> > for these remaining metrics are all zero.
> > >> >
> > >> > I saw that there was some refactoring of metric naming code. Was the
> > >> > behavior supposed to have changed?
> > >> >
> > >> > Many thanks in advance.
> > >> >
> > >>
> > >
> > >
> >
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Manikumar Reddy <ku...@nmsworks.co.in>.
I have enabled yammer's ConsoleReporter and I am getting all the metrics
(including per-topic metrics).
Yammer's MetricName object implements equals/hashcode methods using
mBeanName . We are constructing a unique mBeanName for each metric, So we
are not missing/overwriting any metrics.
Current confusion is due to MetricName.name(). This will be same
(BytesInPerSec) for both broker level and topic level metrics. We need to
use MetricName.getMBeanName() to differentiate between broker level and
topic level metrics.
0.8.1 MBeanName:
"kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec"
"kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec"
0.8.2 MBeanName:
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC
ConsoleReporter's O/P:
BytesInPerSec: <- This is broker level
count = 1521
mean rate = 3.63 bytes/s
1-minute rate = 0.35 bytes/s
5-minute rate = 2.07 bytes/s
15-minute rate = 1.25 bytes/s
BytesInPerSec: <- This is for topic1
count = 626
mean rate = 1.89 bytes/s
1-minute rate = 0.42 bytes/s
5-minute rate = 31.53 bytes/s
15-minute rate = 64.66 bytes/s
BytesInPerSec: <- This is for topic2
count = 895
mean rate = 3.62 bytes/s
1-minute rate = 1.39 bytes/s
5-minute rate = 30.08 bytes/s
15-minute rate = 50.27 bytes/s
Manikumar
On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> Ok,
>
> It looks like the yammer MetricName is not being created correctly for the
> sub metrics that include a topic. E.g. a metric with an mbeanName like:
>
> kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
>
> appears to be malformed. A yammer MetricName has 4 fields that are used in
> creating a graphite metric, that are included in the constructor:
> group, type, name, scope.
>
> In this case, the metric with the above mbeanName has these fields set in
> the MetricName:
>
> group: "kafka.server"
> type: "BrokerTopicMetrics"
> name: "BytesInPerSec"
> scope: null
>
> Thus, the topic metrics all look the same, and get lumped into the
> top-level BrokerTopicMetrics (and thus that will now be double counted). It
> looks like the fix for kafka-1481 was where things got broken. It seems to
> have introduced ‘tags’ in the building of metric names, and then those tags
> only get applied to the mbeanName, but get excluded from the metric name:
>
> https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
>
> This is a pretty severe issue, since the yammer metrics for these stats
> will be double counted in aggregate, and the per-topic stats will be
> removed.
>
> I should note too, in my previous email, I thought that only the per-topic
> BrokerTopicMetrics were missing, but also several other per-topic metrics
> are missing too, e.g. under kafka.log, etc.
>
> Jason
>
>
> On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <jb...@squareup.com> wrote:
>
> > I can confirm that the per topic metrics are not coming through to the
> > yammer metrics registry. I do see them in jmx (via jconsole), but the
> > MetricsRegistry does not have them.
> > All the other metrics are coming through that appear in jmx.
> >
> > This is with single node instance running locally.
> >
> > Jason
> >
> >
> >
> > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <ku...@nmsworks.co.in>
> > wrote:
> >
> >> If you are using multi-node cluster, then metrics may be reported from
> >> other servers.
> >> pl check all the servers in the cluster.
> >>
> >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <ky...@gmail.com>
> >> wrote:
> >>
> >> > I've been using a custom KafkaMetricsReporter to report Kafka broker
> >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
> >> messages in
> >> > and out for all topics together and for each individual topic.
> >> >
> >> > After upgrading to v0.8.2.0, these metrics are no longer being
> reported.
> >> >
> >> > I'm only seeing the following:
> >> > BrokerTopicMetrics
> >> > - BytesInPerSec
> >> > - BytesOutPerSec
> >> > - BytesRejectedPerSec
> >> > - MessagesInPerSec
> >> >
> >> > What's more, despite lots of successful writes to the cluster, the
> >> values
> >> > for these remaining metrics are all zero.
> >> >
> >> > I saw that there was some refactoring of metric naming code. Was the
> >> > behavior supposed to have changed?
> >> >
> >> > Many thanks in advance.
> >> >
> >>
> >
> >
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Jason Rosenberg <jb...@squareup.com>.
Ok,
It looks like the yammer MetricName is not being created correctly for the
sub metrics that include a topic. E.g. a metric with an mbeanName like:
kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
appears to be malformed. A yammer MetricName has 4 fields that are used in
creating a graphite metric, that are included in the constructor:
group, type, name, scope.
In this case, the metric with the above mbeanName has these fields set in
the MetricName:
group: "kafka.server"
type: "BrokerTopicMetrics"
name: "BytesInPerSec"
scope: null
Thus, the topic metrics all look the same, and get lumped into the
top-level BrokerTopicMetrics (and thus that will now be double counted). It
looks like the fix for kafka-1481 was where things got broken. It seems to
have introduced ‘tags’ in the building of metric names, and then those tags
only get applied to the mbeanName, but get excluded from the metric name:
https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
This is a pretty severe issue, since the yammer metrics for these stats
will be double counted in aggregate, and the per-topic stats will be
removed.
I should note too, in my previous email, I thought that only the per-topic
BrokerTopicMetrics were missing, but also several other per-topic metrics
are missing too, e.g. under kafka.log, etc.
Jason
On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <jb...@squareup.com> wrote:
> I can confirm that the per topic metrics are not coming through to the
> yammer metrics registry. I do see them in jmx (via jconsole), but the
> MetricsRegistry does not have them.
> All the other metrics are coming through that appear in jmx.
>
> This is with single node instance running locally.
>
> Jason
>
>
>
> On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <ku...@nmsworks.co.in>
> wrote:
>
>> If you are using multi-node cluster, then metrics may be reported from
>> other servers.
>> pl check all the servers in the cluster.
>>
>> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <ky...@gmail.com>
>> wrote:
>>
>> > I've been using a custom KafkaMetricsReporter to report Kafka broker
>> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
>> messages in
>> > and out for all topics together and for each individual topic.
>> >
>> > After upgrading to v0.8.2.0, these metrics are no longer being reported.
>> >
>> > I'm only seeing the following:
>> > BrokerTopicMetrics
>> > - BytesInPerSec
>> > - BytesOutPerSec
>> > - BytesRejectedPerSec
>> > - MessagesInPerSec
>> >
>> > What's more, despite lots of successful writes to the cluster, the
>> values
>> > for these remaining metrics are all zero.
>> >
>> > I saw that there was some refactoring of metric naming code. Was the
>> > behavior supposed to have changed?
>> >
>> > Many thanks in advance.
>> >
>>
>
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Joel Koshy <jj...@gmail.com>.
Hi Jason - can you describe how you verify that the metrics are not
coming through to the metrics registry? Looking at the metrics code
it seems that the mbeans are only registered by the yammer jmx
reporter only after being added to the metrics registry.
Thanks,
Joel
On Tue, Jan 27, 2015 at 02:20:38AM -0500, Jason Rosenberg wrote:
> I can confirm that the per topic metrics are not coming through to the
> yammer metrics registry. I do see them in jmx (via jconsole), but the
> MetricsRegistry does not have them.
> All the other metrics are coming through that appear in jmx.
>
> This is with single node instance running locally.
>
> Jason
>
>
>
> On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <ku...@nmsworks.co.in>
> wrote:
>
> > If you are using multi-node cluster, then metrics may be reported from
> > other servers.
> > pl check all the servers in the cluster.
> >
> > On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <ky...@gmail.com> wrote:
> >
> > > I've been using a custom KafkaMetricsReporter to report Kafka broker
> > > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages
> > in
> > > and out for all topics together and for each individual topic.
> > >
> > > After upgrading to v0.8.2.0, these metrics are no longer being reported.
> > >
> > > I'm only seeing the following:
> > > BrokerTopicMetrics
> > > - BytesInPerSec
> > > - BytesOutPerSec
> > > - BytesRejectedPerSec
> > > - MessagesInPerSec
> > >
> > > What's more, despite lots of successful writes to the cluster, the values
> > > for these remaining metrics are all zero.
> > >
> > > I saw that there was some refactoring of metric naming code. Was the
> > > behavior supposed to have changed?
> > >
> > > Many thanks in advance.
> > >
> >
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Jason Rosenberg <jb...@squareup.com>.
I can confirm that the per topic metrics are not coming through to the
yammer metrics registry. I do see them in jmx (via jconsole), but the
MetricsRegistry does not have them.
All the other metrics are coming through that appear in jmx.
This is with single node instance running locally.
Jason
On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <ku...@nmsworks.co.in>
wrote:
> If you are using multi-node cluster, then metrics may be reported from
> other servers.
> pl check all the servers in the cluster.
>
> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <ky...@gmail.com> wrote:
>
> > I've been using a custom KafkaMetricsReporter to report Kafka broker
> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages
> in
> > and out for all topics together and for each individual topic.
> >
> > After upgrading to v0.8.2.0, these metrics are no longer being reported.
> >
> > I'm only seeing the following:
> > BrokerTopicMetrics
> > - BytesInPerSec
> > - BytesOutPerSec
> > - BytesRejectedPerSec
> > - MessagesInPerSec
> >
> > What's more, despite lots of successful writes to the cluster, the values
> > for these remaining metrics are all zero.
> >
> > I saw that there was some refactoring of metric naming code. Was the
> > behavior supposed to have changed?
> >
> > Many thanks in advance.
> >
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Manikumar Reddy <ku...@nmsworks.co.in>.
If you are using multi-node cluster, then metrics may be reported from
other servers.
pl check all the servers in the cluster.
On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <ky...@gmail.com> wrote:
> I've been using a custom KafkaMetricsReporter to report Kafka broker
> metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages in
> and out for all topics together and for each individual topic.
>
> After upgrading to v0.8.2.0, these metrics are no longer being reported.
>
> I'm only seeing the following:
> BrokerTopicMetrics
> - BytesInPerSec
> - BytesOutPerSec
> - BytesRejectedPerSec
> - MessagesInPerSec
>
> What's more, despite lots of successful writes to the cluster, the values
> for these remaining metrics are all zero.
>
> I saw that there was some refactoring of metric naming code. Was the
> behavior supposed to have changed?
>
> Many thanks in advance.
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Jun Rao <ju...@confluent.io>.
The jmx beans created by Yammer have been changed to the new names.
However, the global mbeans have different names from the topic level
mbeans. The new mbeans have completely different names from the old ones
since they no longer have quotes. Does that answer your question?
Thanks,
Jun
On Mon, Jan 26, 2015 at 3:38 PM, Kyle Banker <ky...@gmail.com> wrote:
> Thanks, Jun. I'm pretty sure, though not 100% confident, that this has
> caused a change in the way that these metrics are reported to Yammer
> metrics (I believe that they are stomping on each other). Is that intended
> as well?
>
> On Mon, Jan 26, 2015 at 4:13 PM, Jun Rao <ju...@confluent.io> wrote:
>
> > Yes, we refactored the mbean names in 0.8.2.0 to make them more standard.
> > Those metrics are now registered under the following names. I did some
> > quick test and the values do get updated.
> >
> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=test
> >
> > The full list of the refactored mbean names can be found at
> > http://kafka.apache.org/082/documentation.html#monitoring
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Jan 26, 2015 at 2:42 PM, Kyle Banker <ky...@gmail.com>
> wrote:
> >
> > > I've been using a custom KafkaMetricsReporter to report Kafka broker
> > > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
> messages
> > in
> > > and out for all topics together and for each individual topic.
> > >
> > > After upgrading to v0.8.2.0, these metrics are no longer being
> reported.
> > >
> > > I'm only seeing the following:
> > > BrokerTopicMetrics
> > > - BytesInPerSec
> > > - BytesOutPerSec
> > > - BytesRejectedPerSec
> > > - MessagesInPerSec
> > >
> > > What's more, despite lots of successful writes to the cluster, the
> values
> > > for these remaining metrics are all zero.
> > >
> > > I saw that there was some refactoring of metric naming code. Was the
> > > behavior supposed to have changed?
> > >
> > > Many thanks in advance.
> > >
> >
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Kyle Banker <ky...@gmail.com>.
Thanks, Jun. I'm pretty sure, though not 100% confident, that this has
caused a change in the way that these metrics are reported to Yammer
metrics (I believe that they are stomping on each other). Is that intended
as well?
On Mon, Jan 26, 2015 at 4:13 PM, Jun Rao <ju...@confluent.io> wrote:
> Yes, we refactored the mbean names in 0.8.2.0 to make them more standard.
> Those metrics are now registered under the following names. I did some
> quick test and the values do get updated.
>
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=test
>
> The full list of the refactored mbean names can be found at
> http://kafka.apache.org/082/documentation.html#monitoring
>
> Thanks,
>
> Jun
>
>
> On Mon, Jan 26, 2015 at 2:42 PM, Kyle Banker <ky...@gmail.com> wrote:
>
> > I've been using a custom KafkaMetricsReporter to report Kafka broker
> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages
> in
> > and out for all topics together and for each individual topic.
> >
> > After upgrading to v0.8.2.0, these metrics are no longer being reported.
> >
> > I'm only seeing the following:
> > BrokerTopicMetrics
> > - BytesInPerSec
> > - BytesOutPerSec
> > - BytesRejectedPerSec
> > - MessagesInPerSec
> >
> > What's more, despite lots of successful writes to the cluster, the values
> > for these remaining metrics are all zero.
> >
> > I saw that there was some refactoring of metric naming code. Was the
> > behavior supposed to have changed?
> >
> > Many thanks in advance.
> >
>
Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Posted by Jun Rao <ju...@confluent.io>.
Yes, we refactored the mbean names in 0.8.2.0 to make them more standard.
Those metrics are now registered under the following names. I did some
quick test and the values do get updated.
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=test
The full list of the refactored mbean names can be found at
http://kafka.apache.org/082/documentation.html#monitoring
Thanks,
Jun
On Mon, Jan 26, 2015 at 2:42 PM, Kyle Banker <ky...@gmail.com> wrote:
> I've been using a custom KafkaMetricsReporter to report Kafka broker
> metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages in
> and out for all topics together and for each individual topic.
>
> After upgrading to v0.8.2.0, these metrics are no longer being reported.
>
> I'm only seeing the following:
> BrokerTopicMetrics
> - BytesInPerSec
> - BytesOutPerSec
> - BytesRejectedPerSec
> - MessagesInPerSec
>
> What's more, despite lots of successful writes to the cluster, the values
> for these remaining metrics are all zero.
>
> I saw that there was some refactoring of metric naming code. Was the
> behavior supposed to have changed?
>
> Many thanks in advance.
>