You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vadim Keylis <vk...@gmail.com> on 2013/07/01 20:21:09 UTC
Kafka Important Metrics
Good morning. What is the best way to monitor kafka through jmx or by
enabling kafka.csv.metrics.reporter.enabled?
What are the important metrics in JMX to watch for and/or graph?
What are the important metrics in csv files to watch for and/or graph?
Thanks,
Vadim
Re: Kafka Important Metrics
Posted by Cosmin Lehene <cl...@adobe.com>.
I haven't got the chance to try JMXTrans, although I'm planning to (and
like to hear feedback if there is any).
That being said they seem to focus on performance
from their wiki (https://github.com/jmxtrans/jmxtrans/wiki)
"The JmxTransformer engine is fully multithreaded. You specify the maximum
number of threads that you want to start up for each part of the
application. By default, up to 10 servers are queried at the same time. It
is also possible to have multiple threads for each query against a server.
Thus, you can specify that you want 10 threads to handle your 50 servers.
Each one of your servers may have defined 10 queries. You can therefore,
set the numQueryThreads to 2 to execute two queries against a server at
the same time."
Cosmin
On 7/3/13 7:07 PM, "David DeMaagd" <dd...@linkedin.com> wrote:
>I've also used jolokia, http://jolokia.org/, though it can get a little
>slow
>to respond if you don't use it right. Have rolled a JMX/HTTP 'data
>dumper'
>from scratch (can be done in a couple hundred lines of Java without too
>much issue)...
>
>--
>Dave DeMaagd
>ddemaagd@linkedin.com | 818 262 7958
>
>(clehene@adobe.com - Wed, Jul 03, 2013 at 04:22:38PM +0100)
>> I believe something like https://github.com/jmxtrans/jmxtrans could
>>solve
>> this without the need to implement 3rd party monitoring integrations.
>>
>> On 7/2/13 12:57 AM, "Maxime Brugidou" <ma...@gmail.com> wrote:
>>
>> >By the way, having an official contrib package with graphite, ganglia
>>and
>> >other well-known reporters would be awesome so that not everyone has to
>> >write their own.
>> >On Jul 1, 2013 10:27 PM, "Joel Koshy" <jj...@gmail.com> wrote:
>> >
>> >> Also, there are several key metrics on the broker and client side -
>>we
>> >> should compile a list and put it on a wiki. Can you file a jira for
>> >> this?
>> >>
>> >> On Mon, Jul 1, 2013 at 1:26 PM, Joel Koshy <jj...@gmail.com>
>>wrote:
>> >> > CSVreporter is probably not an ideal fit for production monitoring
>>-
>> >> > we use it for getting stats out of period system test runs.
>> >> >
>> >> > For production monitoring you are probably better off reading off
>>JMX
>> >> > and feeding your monitoring system of choice. You can also write a
>> >> > custom metrics reporter and additionally implement
>> >> > KafkaMetricsReporter (see KafkaCSVMetricsReporter.scala for an
>> >> > example) and plug it in the KafkaConfig. Your custom metrics
>>reporter
>> >> > can directly feed into your monitoring system. E.g., if you use
>> >> > graphite, you can write a KafkaGraphiteReporter that wraps the
>> >> > GraphiteReporter that coda hale metrics provides.
>> >> >
>> >> > Joel
>> >> >
>> >> > On Mon, Jul 1, 2013 at 11:21 AM, Vadim Keylis
>><vk...@gmail.com>
>> >> wrote:
>> >> >> Good morning. What is the best way to monitor kafka through jmx
>>or by
>> >> >> enabling kafka.csv.metrics.reporter.enabled?
>> >> >> What are the important metrics in JMX to watch for and/or graph?
>> >> >> What are the important metrics in csv files to watch for and/or
>> >>graph?
>> >> >>
>> >> >> Thanks,
>> >> >> Vadim
>> >>
>>
Re: Kafka Important Metrics
Posted by David DeMaagd <dd...@linkedin.com>.
I've also used jolokia, http://jolokia.org/, though it can get a little slow
to respond if you don't use it right. Have rolled a JMX/HTTP 'data dumper'
from scratch (can be done in a couple hundred lines of Java without too
much issue)...
--
Dave DeMaagd
ddemaagd@linkedin.com | 818 262 7958
(clehene@adobe.com - Wed, Jul 03, 2013 at 04:22:38PM +0100)
> I believe something like https://github.com/jmxtrans/jmxtrans could solve
> this without the need to implement 3rd party monitoring integrations.
>
> On 7/2/13 12:57 AM, "Maxime Brugidou" <ma...@gmail.com> wrote:
>
> >By the way, having an official contrib package with graphite, ganglia and
> >other well-known reporters would be awesome so that not everyone has to
> >write their own.
> >On Jul 1, 2013 10:27 PM, "Joel Koshy" <jj...@gmail.com> wrote:
> >
> >> Also, there are several key metrics on the broker and client side - we
> >> should compile a list and put it on a wiki. Can you file a jira for
> >> this?
> >>
> >> On Mon, Jul 1, 2013 at 1:26 PM, Joel Koshy <jj...@gmail.com> wrote:
> >> > CSVreporter is probably not an ideal fit for production monitoring -
> >> > we use it for getting stats out of period system test runs.
> >> >
> >> > For production monitoring you are probably better off reading off JMX
> >> > and feeding your monitoring system of choice. You can also write a
> >> > custom metrics reporter and additionally implement
> >> > KafkaMetricsReporter (see KafkaCSVMetricsReporter.scala for an
> >> > example) and plug it in the KafkaConfig. Your custom metrics reporter
> >> > can directly feed into your monitoring system. E.g., if you use
> >> > graphite, you can write a KafkaGraphiteReporter that wraps the
> >> > GraphiteReporter that coda hale metrics provides.
> >> >
> >> > Joel
> >> >
> >> > On Mon, Jul 1, 2013 at 11:21 AM, Vadim Keylis <vk...@gmail.com>
> >> wrote:
> >> >> Good morning. What is the best way to monitor kafka through jmx or by
> >> >> enabling kafka.csv.metrics.reporter.enabled?
> >> >> What are the important metrics in JMX to watch for and/or graph?
> >> >> What are the important metrics in csv files to watch for and/or
> >>graph?
> >> >>
> >> >> Thanks,
> >> >> Vadim
> >>
>
Re: Kafka Important Metrics
Posted by Cosmin Lehene <cl...@adobe.com>.
I believe something like https://github.com/jmxtrans/jmxtrans could solve
this without the need to implement 3rd party monitoring integrations.
On 7/2/13 12:57 AM, "Maxime Brugidou" <ma...@gmail.com> wrote:
>By the way, having an official contrib package with graphite, ganglia and
>other well-known reporters would be awesome so that not everyone has to
>write their own.
>On Jul 1, 2013 10:27 PM, "Joel Koshy" <jj...@gmail.com> wrote:
>
>> Also, there are several key metrics on the broker and client side - we
>> should compile a list and put it on a wiki. Can you file a jira for
>> this?
>>
>> On Mon, Jul 1, 2013 at 1:26 PM, Joel Koshy <jj...@gmail.com> wrote:
>> > CSVreporter is probably not an ideal fit for production monitoring -
>> > we use it for getting stats out of period system test runs.
>> >
>> > For production monitoring you are probably better off reading off JMX
>> > and feeding your monitoring system of choice. You can also write a
>> > custom metrics reporter and additionally implement
>> > KafkaMetricsReporter (see KafkaCSVMetricsReporter.scala for an
>> > example) and plug it in the KafkaConfig. Your custom metrics reporter
>> > can directly feed into your monitoring system. E.g., if you use
>> > graphite, you can write a KafkaGraphiteReporter that wraps the
>> > GraphiteReporter that coda hale metrics provides.
>> >
>> > Joel
>> >
>> > On Mon, Jul 1, 2013 at 11:21 AM, Vadim Keylis <vk...@gmail.com>
>> wrote:
>> >> Good morning. What is the best way to monitor kafka through jmx or by
>> >> enabling kafka.csv.metrics.reporter.enabled?
>> >> What are the important metrics in JMX to watch for and/or graph?
>> >> What are the important metrics in csv files to watch for and/or
>>graph?
>> >>
>> >> Thanks,
>> >> Vadim
>>
Re: Kafka Important Metrics
Posted by Hanish Bansal <ha...@gmail.com>.
Hi Vadim,
It totally depends on your requirement. If you want all metrics in some
file then other option may be better otherwise the best way to monitor
kafka is through jmx.
Also list of metrics varies in different Kafka versions. I am assuming that
you are using recent Kafka release of 0.8.
Some important metrics in in Kafka-0.8 through JMX are:
BytesInCount
BytesOutCount
BytesInPerSecond
BytesOutPerSecond
FailedProduceRequestsCount
FailedProduceRequestsPerSecond
FailedFetchRequestsCount
FailedFetchRequestsPerSecond
MessageInCount
MessageInPerSecond
Hope it will be helpful for you.
On Mon, Jul 1, 2013 at 9:57 PM, Maxime Brugidou
<ma...@gmail.com>wrote:
> By the way, having an official contrib package with graphite, ganglia and
> other well-known reporters would be awesome so that not everyone has to
> write their own.
> On Jul 1, 2013 10:27 PM, "Joel Koshy" <jj...@gmail.com> wrote:
>
> > Also, there are several key metrics on the broker and client side - we
> > should compile a list and put it on a wiki. Can you file a jira for
> > this?
> >
> > On Mon, Jul 1, 2013 at 1:26 PM, Joel Koshy <jj...@gmail.com> wrote:
> > > CSVreporter is probably not an ideal fit for production monitoring -
> > > we use it for getting stats out of period system test runs.
> > >
> > > For production monitoring you are probably better off reading off JMX
> > > and feeding your monitoring system of choice. You can also write a
> > > custom metrics reporter and additionally implement
> > > KafkaMetricsReporter (see KafkaCSVMetricsReporter.scala for an
> > > example) and plug it in the KafkaConfig. Your custom metrics reporter
> > > can directly feed into your monitoring system. E.g., if you use
> > > graphite, you can write a KafkaGraphiteReporter that wraps the
> > > GraphiteReporter that coda hale metrics provides.
> > >
> > > Joel
> > >
> > > On Mon, Jul 1, 2013 at 11:21 AM, Vadim Keylis <vk...@gmail.com>
> > wrote:
> > >> Good morning. What is the best way to monitor kafka through jmx or by
> > >> enabling kafka.csv.metrics.reporter.enabled?
> > >> What are the important metrics in JMX to watch for and/or graph?
> > >> What are the important metrics in csv files to watch for and/or graph?
> > >>
> > >> Thanks,
> > >> Vadim
> >
>
--
*Thanks & Regards*
*Hanish Bansal*
Re: Kafka Important Metrics
Posted by Maxime Brugidou <ma...@gmail.com>.
By the way, having an official contrib package with graphite, ganglia and
other well-known reporters would be awesome so that not everyone has to
write their own.
On Jul 1, 2013 10:27 PM, "Joel Koshy" <jj...@gmail.com> wrote:
> Also, there are several key metrics on the broker and client side - we
> should compile a list and put it on a wiki. Can you file a jira for
> this?
>
> On Mon, Jul 1, 2013 at 1:26 PM, Joel Koshy <jj...@gmail.com> wrote:
> > CSVreporter is probably not an ideal fit for production monitoring -
> > we use it for getting stats out of period system test runs.
> >
> > For production monitoring you are probably better off reading off JMX
> > and feeding your monitoring system of choice. You can also write a
> > custom metrics reporter and additionally implement
> > KafkaMetricsReporter (see KafkaCSVMetricsReporter.scala for an
> > example) and plug it in the KafkaConfig. Your custom metrics reporter
> > can directly feed into your monitoring system. E.g., if you use
> > graphite, you can write a KafkaGraphiteReporter that wraps the
> > GraphiteReporter that coda hale metrics provides.
> >
> > Joel
> >
> > On Mon, Jul 1, 2013 at 11:21 AM, Vadim Keylis <vk...@gmail.com>
> wrote:
> >> Good morning. What is the best way to monitor kafka through jmx or by
> >> enabling kafka.csv.metrics.reporter.enabled?
> >> What are the important metrics in JMX to watch for and/or graph?
> >> What are the important metrics in csv files to watch for and/or graph?
> >>
> >> Thanks,
> >> Vadim
>
Re: Kafka Important Metrics
Posted by Vadim Keylis <vk...@gmail.com>.
Good morning Joel. The bug has been filed
https://issues.apache.org/jira/browse/KAFKA-958
On Mon, Jul 1, 2013 at 1:27 PM, Joel Koshy <jj...@gmail.com> wrote:
> Also, there are several key metrics on the broker and client side - we
> should compile a list and put it on a wiki. Can you file a jira for
> this?
>
> On Mon, Jul 1, 2013 at 1:26 PM, Joel Koshy <jj...@gmail.com> wrote:
> > CSVreporter is probably not an ideal fit for production monitoring -
> > we use it for getting stats out of period system test runs.
> >
> > For production monitoring you are probably better off reading off JMX
> > and feeding your monitoring system of choice. You can also write a
> > custom metrics reporter and additionally implement
> > KafkaMetricsReporter (see KafkaCSVMetricsReporter.scala for an
> > example) and plug it in the KafkaConfig. Your custom metrics reporter
> > can directly feed into your monitoring system. E.g., if you use
> > graphite, you can write a KafkaGraphiteReporter that wraps the
> > GraphiteReporter that coda hale metrics provides.
> >
> > Joel
> >
> > On Mon, Jul 1, 2013 at 11:21 AM, Vadim Keylis <vk...@gmail.com>
> wrote:
> >> Good morning. What is the best way to monitor kafka through jmx or by
> >> enabling kafka.csv.metrics.reporter.enabled?
> >> What are the important metrics in JMX to watch for and/or graph?
> >> What are the important metrics in csv files to watch for and/or graph?
> >>
> >> Thanks,
> >> Vadim
>
Re: Kafka Important Metrics
Posted by Joel Koshy <jj...@gmail.com>.
Also, there are several key metrics on the broker and client side - we
should compile a list and put it on a wiki. Can you file a jira for
this?
On Mon, Jul 1, 2013 at 1:26 PM, Joel Koshy <jj...@gmail.com> wrote:
> CSVreporter is probably not an ideal fit for production monitoring -
> we use it for getting stats out of period system test runs.
>
> For production monitoring you are probably better off reading off JMX
> and feeding your monitoring system of choice. You can also write a
> custom metrics reporter and additionally implement
> KafkaMetricsReporter (see KafkaCSVMetricsReporter.scala for an
> example) and plug it in the KafkaConfig. Your custom metrics reporter
> can directly feed into your monitoring system. E.g., if you use
> graphite, you can write a KafkaGraphiteReporter that wraps the
> GraphiteReporter that coda hale metrics provides.
>
> Joel
>
> On Mon, Jul 1, 2013 at 11:21 AM, Vadim Keylis <vk...@gmail.com> wrote:
>> Good morning. What is the best way to monitor kafka through jmx or by
>> enabling kafka.csv.metrics.reporter.enabled?
>> What are the important metrics in JMX to watch for and/or graph?
>> What are the important metrics in csv files to watch for and/or graph?
>>
>> Thanks,
>> Vadim
Re: Kafka Important Metrics
Posted by Joel Koshy <jj...@gmail.com>.
CSVreporter is probably not an ideal fit for production monitoring -
we use it for getting stats out of period system test runs.
For production monitoring you are probably better off reading off JMX
and feeding your monitoring system of choice. You can also write a
custom metrics reporter and additionally implement
KafkaMetricsReporter (see KafkaCSVMetricsReporter.scala for an
example) and plug it in the KafkaConfig. Your custom metrics reporter
can directly feed into your monitoring system. E.g., if you use
graphite, you can write a KafkaGraphiteReporter that wraps the
GraphiteReporter that coda hale metrics provides.
Joel
On Mon, Jul 1, 2013 at 11:21 AM, Vadim Keylis <vk...@gmail.com> wrote:
> Good morning. What is the best way to monitor kafka through jmx or by
> enabling kafka.csv.metrics.reporter.enabled?
> What are the important metrics in JMX to watch for and/or graph?
> What are the important metrics in csv files to watch for and/or graph?
>
> Thanks,
> Vadim