You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Arnaud Lawson <ar...@velos.io> on 2014/03/31 17:36:51 UTC
Kafka Queue Depth Metrics
Hi,
Does anyone know what the JMX metric(s) are for determining the depth of
the kafka queue? I basically want to know the amount of messages that are
on the queue at a certain point in time. Please let me know if there is a
way to find that out. Thanks.
--
Arnaud Lawson
,
Systems Operations Engineer
*VELOS*
Accelerating Machine Learning.
440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
C: (240) 393 - 6703
F: 646.349.4063
E: arnaud.lawson@velos.io
W: www.velos.io
Re: Kafka Queue Depth Metrics
Posted by Arnaud Lawson <ar...@velos.io>.
Ok thank you.
Arnaud-
On Tue, Apr 1, 2014 at 10:50 AM, Jun Rao <ju...@gmail.com> wrote:
> Search for MaxLag in http://kafka.apache.org/documentation.html#monitoring
> .
> Note that this is only available in 0.8.0 and above.
>
> Thanks,
>
> Jun
>
>
> On Tue, Apr 1, 2014 at 7:21 AM, Arnaud Lawson <arnaud.lawson@velos.io
> >wrote:
>
> > Thanks Jun,
> >
> > So on the consumer side is the metric (measuring the # of messages a
> > consumer is behind) called getOffsetLag?
> >
> > Reference:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
> >
> >
> > Arnaud-
> >
> >
> > On Tue, Apr 1, 2014 at 12:16 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > We also have a metric in the consumer that measures the # of messages a
> > > consumer is behind.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Mar 31, 2014 at 8:36 AM, Arnaud Lawson <arnaud.lawson@velos.io
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > Does anyone know what the JMX metric(s) are for determining the depth
> > of
> > > > the kafka queue? I basically want to know the amount of messages that
> > are
> > > > on the queue at a certain point in time. Please let me know if there
> > is a
> > > > way to find that out. Thanks.
> > > >
> > > > --
> > > >
> > > > Arnaud Lawson
> > > > ,
> > > >
> > > > Systems Operations Engineer
> > > > *VELOS*
> > > > Accelerating Machine Learning.
> > > >
> > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> > > > C: (240) 393 - 6703
> > > > F: 646.349.4063
> > > > E: arnaud.lawson@velos.io
> > > > W: www.velos.io
> > > >
> > >
> >
> >
> >
> > --
> >
> > Arnaud Lawson
> > ,
> >
> > Systems Operations Engineer
> > *VELOS*
> > Accelerating Machine Learning.
> >
> > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> > C: (240) 393 - 6703
> > F: 646.349.4063
> > E: arnaud.lawson@velos.io
> > W: www.velos.io
> >
>
--
Arnaud Lawson
,
Systems Operations Engineer
*VELOS*
Accelerating Machine Learning.
440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
C: (240) 393 - 6703
F: 646.349.4063
E: arnaud.lawson@velos.io
W: www.velos.io
Re: Kafka Queue Depth Metrics
Posted by Jun Rao <ju...@gmail.com>.
Search for MaxLag in http://kafka.apache.org/documentation.html#monitoring.
Note that this is only available in 0.8.0 and above.
Thanks,
Jun
On Tue, Apr 1, 2014 at 7:21 AM, Arnaud Lawson <ar...@velos.io>wrote:
> Thanks Jun,
>
> So on the consumer side is the metric (measuring the # of messages a
> consumer is behind) called getOffsetLag?
>
> Reference:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>
>
> Arnaud-
>
>
> On Tue, Apr 1, 2014 at 12:16 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > We also have a metric in the consumer that measures the # of messages a
> > consumer is behind.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Mar 31, 2014 at 8:36 AM, Arnaud Lawson <arnaud.lawson@velos.io
> > >wrote:
> >
> > > Hi,
> > >
> > > Does anyone know what the JMX metric(s) are for determining the depth
> of
> > > the kafka queue? I basically want to know the amount of messages that
> are
> > > on the queue at a certain point in time. Please let me know if there
> is a
> > > way to find that out. Thanks.
> > >
> > > --
> > >
> > > Arnaud Lawson
> > > ,
> > >
> > > Systems Operations Engineer
> > > *VELOS*
> > > Accelerating Machine Learning.
> > >
> > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> > > C: (240) 393 - 6703
> > > F: 646.349.4063
> > > E: arnaud.lawson@velos.io
> > > W: www.velos.io
> > >
> >
>
>
>
> --
>
> Arnaud Lawson
> ,
>
> Systems Operations Engineer
> *VELOS*
> Accelerating Machine Learning.
>
> 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> C: (240) 393 - 6703
> F: 646.349.4063
> E: arnaud.lawson@velos.io
> W: www.velos.io
>
Re: Kafka Queue Depth Metrics
Posted by Arnaud Lawson <ar...@velos.io>.
Thanks Jun,
So on the consumer side is the metric (measuring the # of messages a
consumer is behind) called getOffsetLag?
Reference:
https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
Arnaud-
On Tue, Apr 1, 2014 at 12:16 AM, Jun Rao <ju...@gmail.com> wrote:
> We also have a metric in the consumer that measures the # of messages a
> consumer is behind.
>
> Thanks,
>
> Jun
>
>
> On Mon, Mar 31, 2014 at 8:36 AM, Arnaud Lawson <arnaud.lawson@velos.io
> >wrote:
>
> > Hi,
> >
> > Does anyone know what the JMX metric(s) are for determining the depth of
> > the kafka queue? I basically want to know the amount of messages that are
> > on the queue at a certain point in time. Please let me know if there is a
> > way to find that out. Thanks.
> >
> > --
> >
> > Arnaud Lawson
> > ,
> >
> > Systems Operations Engineer
> > *VELOS*
> > Accelerating Machine Learning.
> >
> > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> > C: (240) 393 - 6703
> > F: 646.349.4063
> > E: arnaud.lawson@velos.io
> > W: www.velos.io
> >
>
--
Arnaud Lawson
,
Systems Operations Engineer
*VELOS*
Accelerating Machine Learning.
440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
C: (240) 393 - 6703
F: 646.349.4063
E: arnaud.lawson@velos.io
W: www.velos.io
Re: Kafka Queue Depth Metrics
Posted by Jun Rao <ju...@gmail.com>.
We also have a metric in the consumer that measures the # of messages a
consumer is behind.
Thanks,
Jun
On Mon, Mar 31, 2014 at 8:36 AM, Arnaud Lawson <ar...@velos.io>wrote:
> Hi,
>
> Does anyone know what the JMX metric(s) are for determining the depth of
> the kafka queue? I basically want to know the amount of messages that are
> on the queue at a certain point in time. Please let me know if there is a
> way to find that out. Thanks.
>
> --
>
> Arnaud Lawson
> ,
>
> Systems Operations Engineer
> *VELOS*
> Accelerating Machine Learning.
>
> 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> C: (240) 393 - 6703
> F: 646.349.4063
> E: arnaud.lawson@velos.io
> W: www.velos.io
>
Re: Kafka Queue Depth Metrics
Posted by Chi Hoang <ch...@groupon.com>.
We've used bytes written as in, and bytes read as out.
Chi
On Mon, Mar 31, 2014 at 11:12 AM, Arnaud Lawson <ar...@velos.io>wrote:
> Ok thanks Guozhang . So What would the metrics for the in-traffic and
> out-traffic message count be on this page (this page shows all existing jmx
> metrics)?
>
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>
>
> On Mon, Mar 31, 2014 at 1:01 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Arnaud,
> >
> > Currently we do not have metric(s) for the number of messages on the
> kafka
> > broker, though we do have the metrics for the in-traffic and out-traffic
> > message count.
> >
> > One work-around would be the ConsumerOffsetChecker tool, for which it
> shows
> > the last offset of messages on the brokers v.s. the consumed last offsets
> > of the specified consumer on certain topics.
> >
> > Guozhang
> >
> >
> > On Mon, Mar 31, 2014 at 8:36 AM, Arnaud Lawson <arnaud.lawson@velos.io
> > >wrote:
> >
> > > Hi,
> > >
> > > Does anyone know what the JMX metric(s) are for determining the depth
> of
> > > the kafka queue? I basically want to know the amount of messages that
> are
> > > on the queue at a certain point in time. Please let me know if there
> is a
> > > way to find that out. Thanks.
> > >
> > > --
> > >
> > > Arnaud Lawson
> > > ,
> > >
> > > Systems Operations Engineer
> > > *VELOS*
> > > Accelerating Machine Learning.
> > >
> > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> > > C: (240) 393 - 6703
> > > F: 646.349.4063
> > > E: arnaud.lawson@velos.io
> > > W: www.velos.io
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
>
> Arnaud Lawson
> ,
>
> Systems Operations Engineer
> *VELOS*
> Accelerating Machine Learning.
>
> 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> C: (240) 393 - 6703
> F: 646.349.4063
> E: arnaud.lawson@velos.io
> W: www.velos.io
>
Re: Kafka Queue Depth Metrics
Posted by Guozhang Wang <wa...@gmail.com>.
getBytesIn and getBytesOut in BrokerAllTopicStat
On Mon, Mar 31, 2014 at 11:12 AM, Arnaud Lawson <ar...@velos.io>wrote:
> Ok thanks Guozhang . So What would the metrics for the in-traffic and
> out-traffic message count be on this page (this page shows all existing jmx
> metrics)?
>
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>
>
> On Mon, Mar 31, 2014 at 1:01 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Arnaud,
> >
> > Currently we do not have metric(s) for the number of messages on the
> kafka
> > broker, though we do have the metrics for the in-traffic and out-traffic
> > message count.
> >
> > One work-around would be the ConsumerOffsetChecker tool, for which it
> shows
> > the last offset of messages on the brokers v.s. the consumed last offsets
> > of the specified consumer on certain topics.
> >
> > Guozhang
> >
> >
> > On Mon, Mar 31, 2014 at 8:36 AM, Arnaud Lawson <arnaud.lawson@velos.io
> > >wrote:
> >
> > > Hi,
> > >
> > > Does anyone know what the JMX metric(s) are for determining the depth
> of
> > > the kafka queue? I basically want to know the amount of messages that
> are
> > > on the queue at a certain point in time. Please let me know if there
> is a
> > > way to find that out. Thanks.
> > >
> > > --
> > >
> > > Arnaud Lawson
> > > ,
> > >
> > > Systems Operations Engineer
> > > *VELOS*
> > > Accelerating Machine Learning.
> > >
> > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> > > C: (240) 393 - 6703
> > > F: 646.349.4063
> > > E: arnaud.lawson@velos.io
> > > W: www.velos.io
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
>
> Arnaud Lawson
> ,
>
> Systems Operations Engineer
> *VELOS*
> Accelerating Machine Learning.
>
> 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> C: (240) 393 - 6703
> F: 646.349.4063
> E: arnaud.lawson@velos.io
> W: www.velos.io
>
--
-- Guozhang
Re: Kafka Queue Depth Metrics
Posted by Arnaud Lawson <ar...@velos.io>.
Ok thanks Guozhang . So What would the metrics for the in-traffic and
out-traffic message count be on this page (this page shows all existing jmx
metrics)?
https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
On Mon, Mar 31, 2014 at 1:01 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Arnaud,
>
> Currently we do not have metric(s) for the number of messages on the kafka
> broker, though we do have the metrics for the in-traffic and out-traffic
> message count.
>
> One work-around would be the ConsumerOffsetChecker tool, for which it shows
> the last offset of messages on the brokers v.s. the consumed last offsets
> of the specified consumer on certain topics.
>
> Guozhang
>
>
> On Mon, Mar 31, 2014 at 8:36 AM, Arnaud Lawson <arnaud.lawson@velos.io
> >wrote:
>
> > Hi,
> >
> > Does anyone know what the JMX metric(s) are for determining the depth of
> > the kafka queue? I basically want to know the amount of messages that are
> > on the queue at a certain point in time. Please let me know if there is a
> > way to find that out. Thanks.
> >
> > --
> >
> > Arnaud Lawson
> > ,
> >
> > Systems Operations Engineer
> > *VELOS*
> > Accelerating Machine Learning.
> >
> > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> > C: (240) 393 - 6703
> > F: 646.349.4063
> > E: arnaud.lawson@velos.io
> > W: www.velos.io
> >
>
>
>
> --
> -- Guozhang
>
--
Arnaud Lawson
,
Systems Operations Engineer
*VELOS*
Accelerating Machine Learning.
440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
C: (240) 393 - 6703
F: 646.349.4063
E: arnaud.lawson@velos.io
W: www.velos.io
Re: Kafka Queue Depth Metrics
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Arnaud,
Currently we do not have metric(s) for the number of messages on the kafka
broker, though we do have the metrics for the in-traffic and out-traffic
message count.
One work-around would be the ConsumerOffsetChecker tool, for which it shows
the last offset of messages on the brokers v.s. the consumed last offsets
of the specified consumer on certain topics.
Guozhang
On Mon, Mar 31, 2014 at 8:36 AM, Arnaud Lawson <ar...@velos.io>wrote:
> Hi,
>
> Does anyone know what the JMX metric(s) are for determining the depth of
> the kafka queue? I basically want to know the amount of messages that are
> on the queue at a certain point in time. Please let me know if there is a
> way to find that out. Thanks.
>
> --
>
> Arnaud Lawson
> ,
>
> Systems Operations Engineer
> *VELOS*
> Accelerating Machine Learning.
>
> 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001
> C: (240) 393 - 6703
> F: 646.349.4063
> E: arnaud.lawson@velos.io
> W: www.velos.io
>
--
-- Guozhang