You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Dongjin Lee <do...@apache.org> on 2016/12/27 14:31:01 UTC
Experiencing trouble with KafkaCSVMetricsReporter
In short: the resulting csv files from the brokers are filled with 0 only, although the broker cluster is running correctly.
Hello. I am trying some benchmarks with KAFKA-4514[^1]. However, My KafkaCSVMetricsReporter is not working properly. I would like to ask if someone on this mailing list has experienced similar cases.
Let me explain. I configured a Kafka cluster with 3 zookeeper instances and 3 Kafka Broker instances. After confirming all is working correctly, I added following properties in server.properties file and restarted the brokers.
> kafka.metrics.polling.interval.secs=5> kafka.metrics.reporters=kafka.metrics.KafkaCSVMetricsReporter> kafka.csv.metrics.reporter.enabled=true
By re-running the brokers and the producer, I acquired some csv files from kafka_metrics directory. But there are two weird things:
1. Kafka brokers repeatedly try to create csv files and fail with IO Exception. Why they try to create files, yet the resulting csv files already exist?2. All cells of the resulting csv files are filled with 0, except their headers. Since the producer generated so many messages, they (e.g., BytesPerSec) cannot be 0.
Any pieces of advices or comments are welcome. Thanks in advance.
Regards,Dongjin
[^1]: https://issues.apache.org/jira/browse/KAFKA-4514
Re: Experiencing trouble with KafkaCSVMetricsReporter
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Dongjin,
1. I'm not familiar with `KafkaCSVMetricsReporter`, but it seems to delete
and recreate the csv dir before starting `CsvReporter` which then creates
the csv files. Assuming the permissions are correct, not sure why it would
fail to create the files. It may be worth filing a JIRA (and possibly a PR
with a fix if you figure out the reason).
2. You could check BytesPerSec via JMX to confirm that the problem is
indeed in the CVS reporter (and/or related classes). Again, it's worth
filing a JIRA for this (and a PR with a fix would be even better).
Thanks,
Ismael
On Tue, Dec 27, 2016 at 2:31 PM, Dongjin Lee <do...@apache.org> wrote:
> In short: the resulting csv files from the brokers are filled with 0 only,
> although the broker cluster is running correctly.
>
> Hello. I am trying some benchmarks with KAFKA-4514[^1]. However, My
> KafkaCSVMetricsReporter is not working properly. I would like to ask if
> someone on this mailing list has experienced similar cases.
> Let me explain. I configured a Kafka cluster with 3 zookeeper instances
> and 3 Kafka Broker instances. After confirming all is working correctly, I
> added following properties in server.properties file and restarted the
> brokers.
> > kafka.metrics.polling.interval.secs=5> kafka.metrics.reporters=kafka.
> metrics.KafkaCSVMetricsReporter> kafka.csv.metrics.reporter.enabled=true
> By re-running the brokers and the producer, I acquired some csv files from
> kafka_metrics directory. But there are two weird things:
> 1. Kafka brokers repeatedly try to create csv files and fail with IO
> Exception. Why they try to create files, yet the resulting csv files
> already exist?2. All cells of the resulting csv files are filled with 0,
> except their headers. Since the producer generated so many messages, they
> (e.g., BytesPerSec) cannot be 0.
> Any pieces of advices or comments are welcome. Thanks in advance.
> Regards,Dongjin
> [^1]: https://issues.apache.org/jira/browse/KAFKA-4514
Re: Experiencing trouble with KafkaCSVMetricsReporter
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Dongjin,
1. I'm not familiar with `KafkaCSVMetricsReporter`, but it seems to delete
and recreate the csv dir before starting `CsvReporter` which then creates
the csv files. Assuming the permissions are correct, not sure why it would
fail to create the files. It may be worth filing a JIRA (and possibly a PR
with a fix if you figure out the reason).
2. You could check BytesPerSec via JMX to confirm that the problem is
indeed in the CVS reporter (and/or related classes). Again, it's worth
filing a JIRA for this (and a PR with a fix would be even better).
Thanks,
Ismael
On Tue, Dec 27, 2016 at 2:31 PM, Dongjin Lee <do...@apache.org> wrote:
> In short: the resulting csv files from the brokers are filled with 0 only,
> although the broker cluster is running correctly.
>
> Hello. I am trying some benchmarks with KAFKA-4514[^1]. However, My
> KafkaCSVMetricsReporter is not working properly. I would like to ask if
> someone on this mailing list has experienced similar cases.
> Let me explain. I configured a Kafka cluster with 3 zookeeper instances
> and 3 Kafka Broker instances. After confirming all is working correctly, I
> added following properties in server.properties file and restarted the
> brokers.
> > kafka.metrics.polling.interval.secs=5> kafka.metrics.reporters=kafka.
> metrics.KafkaCSVMetricsReporter> kafka.csv.metrics.reporter.enabled=true
> By re-running the brokers and the producer, I acquired some csv files from
> kafka_metrics directory. But there are two weird things:
> 1. Kafka brokers repeatedly try to create csv files and fail with IO
> Exception. Why they try to create files, yet the resulting csv files
> already exist?2. All cells of the resulting csv files are filled with 0,
> except their headers. Since the producer generated so many messages, they
> (e.g., BytesPerSec) cannot be 0.
> Any pieces of advices or comments are welcome. Thanks in advance.
> Regards,Dongjin
> [^1]: https://issues.apache.org/jira/browse/KAFKA-4514