CSVMetricsReporter name conflict

[Andrew Grasso <ag...@janestreet.com>]

Hello,

I've been trying to monitor a Kafka broker using the CSVMetricsReporter.
However, when I start the broker, only a few csv files are created in the
directory, and then there are repeated IOExceptions in the kafka.out log
stating that CsvReporter.createStreamForMetric cannot create the file
{path_to_metrics_dir}/LocalTimeMs.csv

I can recreate this problem in 0.8.2.1, 0.8.2.2, and 0.9.0.0. CSV reporting
works as expected in 0.8.1.1.

I believe the problem is that in the later versions CSV files are being
named from only the metric name, and not the tags. In 0.8.1.1, the
LocalTimeMs metrics are given a name prefixed with a distinguishing source,
while in later versions this information is passed as a tag. These tags are
passed into the MetricName constructor in
KafkaMetricsGroup.explicitMetricName, but as the mBeanName and not as the
name.

This is causing a name conflict, since CSV files are named off of the name
field, not the mBeanName.

For a similar reason, I cannot get metrics on partition specific data. The
CSV reporter seems to not try any other metrics after raising an exception
on LocalTimeMs. All metrics are published correctly to JMX.

Has anyone encountered this problem? Is there a way to make CSV metrics
work in the stable release (0.8.2.2)?

It is possible I have missed something in following the code, as I am not
very literate in Scala, but I am reasonably confident that my understanding
of the code lines up with the behavior I've seen.

I believe this behavior is related to an old issue, KAFKA-542
<https://issues.apache.org/jira/browse/KAFKA-542>

Best,
Andrew Grasso

Re: CSVMetricsReporter name conflict

[Jun Rao <ju...@confluent.io>]

Andrew,

Yes, in 0.8.2.x, we cleaned up the metric name to put meaning attributes in
tags, instead of in the name. This makes it easier for monitoring
applications to parse those metric names. It seems that we need a
CSVMetricsReporter that can deal with tagged names better. Many people also
use jmxtrans to export the Kafka metrics.

Thanks,

Jun

On Mon, Dec 14, 2015 at 10:32 AM, Andrew Grasso <ag...@janestreet.com>
wrote:

> Hello,
>
> I've been trying to monitor a Kafka broker using the CSVMetricsReporter.
> However, when I start the broker, only a few csv files are created in the
> directory, and then there are repeated IOExceptions in the kafka.out log
> stating that CsvReporter.createStreamForMetric cannot create the file
> {path_to_metrics_dir}/LocalTimeMs.csv
>
> I can recreate this problem in 0.8.2.1, 0.8.2.2, and 0.9.0.0. CSV reporting
> works as expected in 0.8.1.1.
>
> I believe the problem is that in the later versions CSV files are being
> named from only the metric name, and not the tags. In 0.8.1.1, the
> LocalTimeMs metrics are given a name prefixed with a distinguishing source,
> while in later versions this information is passed as a tag. These tags are
> passed into the MetricName constructor in
> KafkaMetricsGroup.explicitMetricName, but as the mBeanName and not as the
> name.
>
> This is causing a name conflict, since CSV files are named off of the name
> field, not the mBeanName.
>
> For a similar reason, I cannot get metrics on partition specific data. The
> CSV reporter seems to not try any other metrics after raising an exception
> on LocalTimeMs. All metrics are published correctly to JMX.
>
> Has anyone encountered this problem? Is there a way to make CSV metrics
> work in the stable release (0.8.2.2)?
>
> It is possible I have missed something in following the code, as I am not
> very literate in Scala, but I am reasonably confident that my understanding
> of the code lines up with the behavior I've seen.
>
> I believe this behavior is related to an old issue, KAFKA-542
> <https://issues.apache.org/jira/browse/KAFKA-542>
>
> Best,
> Andrew Grasso
>