You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jason Rosenberg <jb...@squareup.com> on 2013/10/23 05:55:41 UTC
metrics with generation specific names
I've noticed that there are several metrics that seem useful for monitoring
overtime, but which contain generational timestamps in the metric name.
We are using yammer metrics libraries to send metrics data in a background
thread every 10 seconds (to kafka actually), and then they eventually end
up in a metrics database (graphite, opentsdb). The metrics then get
graphed via UI, and we can see metrics going way back, etc.
Unfortunately, many of the metrics coming from kafka seem to have metric
names that change any time the server or consumer is restarted, which makes
it hard to easily create graphs over long periods of time (spanning app
restarts).
For example:
names like:
kafka.consumer.FetchRequestAndResponseMetrics....
square-1371718712833-e9bb4d10-0-508818741-AllBrokersFetchRequestRateAndTimeMs<https://graphite.vip.sjc1.square/composer/?#>
or:
kafka.consumer.ZookeeperConsumerConnector...topicName.....
square-1373476779391-78aa2e83-0-FetchQueueSize<https://graphite.vip.sjc1.square/composer/?#>
In our staging environment, we have our servers on regular auto-deploy
cycles (they restart every few hours). So just not longitudinally usable
to have metric names constantly changing like this.
Is there something that can easily be done? Is it really necessary to have
so much cryptic info in the metric name?
Jason
Re: metrics with generation specific names
Posted by Jason Rosenberg <jb...@squareup.com>.
filed: https://issues.apache.org/jira/browse/KAFKA-1100
On Wed, Oct 23, 2013 at 12:28 AM, Jun Rao <ju...@gmail.com> wrote:
> Yes, those metrics names could be simplified. Could you file a jira?
>
> Thanks,
>
> Jun
>
>
> On Tue, Oct 22, 2013 at 8:55 PM, Jason Rosenberg <jb...@squareup.com> wrote:
>
> > I've noticed that there are several metrics that seem useful for
> monitoring
> > overtime, but which contain generational timestamps in the metric name.
> >
> > We are using yammer metrics libraries to send metrics data in a
> background
> > thread every 10 seconds (to kafka actually), and then they eventually end
> > up in a metrics database (graphite, opentsdb). The metrics then get
> > graphed via UI, and we can see metrics going way back, etc.
> >
> > Unfortunately, many of the metrics coming from kafka seem to have metric
> > names that change any time the server or consumer is restarted, which
> makes
> > it hard to easily create graphs over long periods of time (spanning app
> > restarts).
> >
> > For example:
> >
> > names like:
> > kafka.consumer.FetchRequestAndResponseMetrics....
> >
> >
> square-1371718712833-e9bb4d10-0-508818741-AllBrokersFetchRequestRateAndTimeMs<
> > https://graphite.vip.sjc1.square/composer/?#>
> >
> > or:
> > kafka.consumer.ZookeeperConsumerConnector...topicName.....
> > square-1373476779391-78aa2e83-0-FetchQueueSize<
> > https://graphite.vip.sjc1.square/composer/?#>
> >
> > In our staging environment, we have our servers on regular auto-deploy
> > cycles (they restart every few hours). So just not longitudinally usable
> > to have metric names constantly changing like this.
> >
> > Is there something that can easily be done? Is it really necessary to
> have
> > so much cryptic info in the metric name?
> >
> > Jason
> >
>
Re: metrics with generation specific names
Posted by Jun Rao <ju...@gmail.com>.
Yes, those metrics names could be simplified. Could you file a jira?
Thanks,
Jun
On Tue, Oct 22, 2013 at 8:55 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> I've noticed that there are several metrics that seem useful for monitoring
> overtime, but which contain generational timestamps in the metric name.
>
> We are using yammer metrics libraries to send metrics data in a background
> thread every 10 seconds (to kafka actually), and then they eventually end
> up in a metrics database (graphite, opentsdb). The metrics then get
> graphed via UI, and we can see metrics going way back, etc.
>
> Unfortunately, many of the metrics coming from kafka seem to have metric
> names that change any time the server or consumer is restarted, which makes
> it hard to easily create graphs over long periods of time (spanning app
> restarts).
>
> For example:
>
> names like:
> kafka.consumer.FetchRequestAndResponseMetrics....
>
> square-1371718712833-e9bb4d10-0-508818741-AllBrokersFetchRequestRateAndTimeMs<
> https://graphite.vip.sjc1.square/composer/?#>
>
> or:
> kafka.consumer.ZookeeperConsumerConnector...topicName.....
> square-1373476779391-78aa2e83-0-FetchQueueSize<
> https://graphite.vip.sjc1.square/composer/?#>
>
> In our staging environment, we have our servers on regular auto-deploy
> cycles (they restart every few hours). So just not longitudinally usable
> to have metric names constantly changing like this.
>
> Is there something that can easily be done? Is it really necessary to have
> so much cryptic info in the metric name?
>
> Jason
>