You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Hu Xi <hu...@hotmail.com> on 2017/11/09 09:03:51 UTC
[DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead
metrics to KafkaConsumer
Hi all,
As per Jun Rao's suggestion, I opened up the KIP-223(https://cwiki.apache.org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning adding new kinds of lag metrics for KafkaConsumer. Be free to leave your comments here. Thanks in advance.
Reply: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Currently, partition-level records-lag metrics would not be displayed under tags 'topic=<topic>,partition=<partition>'. Instead they were shown under client-level due to the incorrect tags used in the code, see below:
/***** Partition level *****/
this.partitionRecordsLag = new MetricNameTemplate("{topic}-{partition}.records-lag", groupName,
"The latest lag of the partition", tags);
this.partitionRecordsLagMax = new MetricNameTemplate("{topic}-{partition}.records-lag-max", groupName,
"The max lag of the partition", tags);
this.partitionRecordsLagAvg = new MetricNameTemplate("{topic}-{partition}.records-lag-avg", groupName,
"The average lag of the partition", tags);
Does it deserve a bug? And if so, will KIP-225 fix it?
________________________________
发件人: charly molter <ch...@gmail.com>
发送时间: 2017年11月15日 20:58
收件人: dev@kafka.apache.org
抄送: Jiangjie Qin
主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi Jun, Jiangle,
I'd just like to clarify that KIP-225 seems to be using per partition
metric the same way as KIP-223 seems to be doing.
I believe avg and max are still necessary because the MetricsReporter
doesn't work in a "push" manner and the "Value" measurableStat will only
keep the last recorded entry.
Therefore a MetricsReporter usually polls to grab a current view with Value
this view is incomplete so it becomes not possible to compute the
Max/Min/Avg.
Max/Min/Avg uses SampledStats which work with a rolling window of samples
and therefore periodic polling would work.
This is why I believe it's necessary to keep Avg, Min and Max for these
metrics as otherwise we wouldn't be able to recompute it in an external
monitoring system.
Am I wrong thinking this?
Thanks,
Charly
On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> Hi, Charly,
>
> Thanks for KIP-225. Your proposal looks reasonable.
>
> Hi, Jiangjie,
>
> Do you think the approach that KIP-225 proposes is better for exposing the
> per partition metric? Also, do we really need the per partition
> record-lag-avg
> and record-lag-max? It seems that an external monitoring system can always
> derive that from the per partition record-lag.
>
> Thanks,
>
> Jun
>
> On Tue, Nov 14, 2017 at 6:57 AM, charly molter <ch...@gmail.com>
> wrote:
>
> > Hi Jun, Hu,
> >
> > I have KIP-225 open for adding tags to records-lag:
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=74686649
> >
> > I have a patch more or less ready so I could probably get the fix checked
> > in (after the vote) and you could build on top of it. Otherwise we could
> > merge both KIPs if you want but they do sound different to me.
> >
> > Thanks!
> > Charly
> >
> > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
> >
> > > Jun,
> > >
> > >
> > > Let me double confirm with your comments:
> > >
> > > 1 remove partition-level records-lead-avg and records-lead-min since
> they
> > > both can be deduced by external monitoring system.
> > >
> > > 2 Tag partition-level records-lead with topic&partition info
> > >
> > >
> > > If they are the case you expect, do we need to do the same thing for
> > those
> > > `lag` metrics? Seems partition-level records-lag metrics are not tagged
> > > with topic&partition information which might deserve a bug.
> > >
> > >
> > > huxihx
> > >
> > >
> > > ________________________________
> > > 发件人: Jun Rao <ju...@confluent.io>
> > > 发送时间: 2017年11月14日 12:44
> > > 收件人: dev@kafka.apache.org
> > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> > > lead metrics to KafkaConsumer
> > >
> > > Hi, Hu,
> > >
> > > Currently, records-lag-max is an attribute for the mbean
> > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > id="{client-id}".
> > > So, it probably makes sense for records-lead-min to be an attribute
> under
> > > the same mbean.
> > >
> > > The partition level records-lead can probably be an attribute for the
> > mbean
> > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > id="{client-id}",topic=topic1,partition=0,
> > > where topic and partition are the tags. This matches the topic level
> > mbeans
> > > that we have in the consumer. I am not sure what the per partition
> level
> > > records-lead-min and records-lead-avg are. Are they the min/avg of the
> > lead
> > > since the consumer is started? I am not sure we need those since an
> > > external monitoring system can always derive them from records-lead.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > >
> > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com> wrote:
> > >
> > > > Jun,
> > > >
> > > > Thanks for the feedback. Some things need to make sure. Currently,
> > these
> > > > new-added metrics follow the exact naming convention with those 'lag'
> > > > counterparts, as shown below:
> > > >
> > > >
> > > > Consumer-level metric:
> > > >
> > > > records-lag-max ==> records-lead-min
> > > >
> > > >
> > > > Partition-level metrics:
> > > >
> > > > <topic>-<partitionId>.records-lag ==>
> <topic>-<partitionId>.
> > > > records-lead
> > > >
> > > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > > records-lead-min
> > > >
> > > > <topic>-<partitionId>.records-lag-avg ==> <topic>-<partitionId>.
> > > > records-lead-avg
> > > >
> > > >
> > > > Correct me if I am wrong, but what you mentioned `*records-lead-avg
> and
> > > > records-lead-min don't need the partition prefix since they are
> > > aggregates
> > > > across all partitions*` seemed to break the naming rule above. Do we
> > > > still have to keep the same rule with the "lag" metrics?
> > > >
> > > >
> > > > huxihx
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------
> > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > *发送时间:* 2017年11月14日 1:48
> > > > *收件人:* dev@kafka.apache.org
> > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> > > > lead metrics to KafkaConsumer
> > > >
> > > > Hi, Hu,
> > > >
> > > > Thanks for the KIP. Looks good overall. Could you document the mbean
> > name
> > > > for the new metrics? We probably want the name to be consistent with
> > > > records-max-lag as described in
> > > > http://kafka.apache.org/documentation/#monitoring. Also, it seems
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> that
> > > [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http:
> > //kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > <http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http:
> > //kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http:
> > //kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > > records-lead-avg and records-lead-min don't need the partition prefix
> > > since
> > > > they are aggregates across all partitions. For records-lead, it seems
> > > that
> > > > it's better to add the topic partition as a tag, instead of as a
> prefix
> > > in
> > > > the metric name.
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > https://cwiki.apache.
> > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning
> > > adding
> > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave your
> > > > comments
> > > > > here. Thanks in advance.
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Charly Molter
> >
>
--
Charly Molter
Re: 答复: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Jun Rao <ju...@confluent.io>.
Hi, Hu,
Yes. Could you start a separate voting thread?
Thanks,
Jun
On Tue, Nov 28, 2017 at 5:02 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi Rao Jun,
>
>
> Already updated the patch per your suggestion and it seems there are no
> further feedbacks on this KIP. Could we vote now?
>
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月23日 6:21
> 收件人: dev@kafka.apache.org
> 主题: Re: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> There are two types of names. One is the sensor name, which has to be
> unique. It's fine if the sensor name includes the topic/partition as the
> prefix since the sensor name is only a string and is not exposed to jmx.
> The second name is the metric name, which will be used in jmx. Currently,
> the existing lag metric name uses topic/partition as the prefix. KIP-225
> tries to change the metric name to use topic/partition as the tag. We can
> just do the same thing for lead by using tags in the metric name.
>
> Thanks,
>
> Jun
>
> On Mon, Nov 20, 2017 at 10:14 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Hi Jun,
> >
> >
> > Seems the prefix that is used to be the unique Sensor name cannot be
> > removed, so should we keep the prefix?
> >
> >
> > ________________________________
> > 发件人: Jun Rao <ju...@confluent.io>
> > 发送时间: 2017年11月21日 3:55
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition lead metrics to KafkaConsumer
> >
> > Hi, Hu,
> >
> > For the new partition level metrics that you are adding, it seems it's
> > better to just add the topic/partition tag instead of using them in the
> > prefix. For the existing lag metrics, we can fix them in KIP-225.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
> >
> > > Jun,
> > >
> > >
> > > Thanks for the comments. Do you think it'd better to add
> topic/partition
> > > tags for those metrics as well as keep the prefix? If those prefixes
> > should
> > > really be removed, does this KIP need to do the same thing for `lag`
> > ones?
> > >
> > > ________________________________
> > > 发件人: Jun Rao <ju...@confluent.io>
> > > 发送时间: 2017年11月18日 8:55
> > > 收件人: dev@kafka.apache.org
> > > 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition lead metrics to KafkaConsumer
> > >
> > > Hi, Charly,
> > >
> > > Thanks for the input. It makes sense.
> > >
> > > Hi, Hu,
> > >
> > > Perhaps we can keep the per partition records-lead-min and
> > records-lead-avg
> > > as you had before, but just add the topic and the partition as the tags
> > > instead of prefix of the metric name.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Wed, Nov 15, 2017 at 4:58 AM, charly molter <
> charly.molter@gmail.com>
> > > wrote:
> > >
> > > > Hi Jun, Jiangle,
> > > >
> > > > I'd just like to clarify that KIP-225 seems to be using per partition
> > > > metric the same way as KIP-223 seems to be doing.
> > > >
> > > > I believe avg and max are still necessary because the MetricsReporter
> > > > doesn't work in a "push" manner and the "Value" measurableStat will
> > only
> > > > keep the last recorded entry.
> > > > Therefore a MetricsReporter usually polls to grab a current view with
> > > Value
> > > > this view is incomplete so it becomes not possible to compute the
> > > > Max/Min/Avg.
> > > > Max/Min/Avg uses SampledStats which work with a rolling window of
> > samples
> > > > and therefore periodic polling would work.
> > > >
> > > > This is why I believe it's necessary to keep Avg, Min and Max for
> these
> > > > metrics as otherwise we wouldn't be able to recompute it in an
> external
> > > > monitoring system.
> > > >
> > > > Am I wrong thinking this?
> > > >
> > > > Thanks,
> > > > Charly
> > > >
> > > >
> > > > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Hi, Charly,
> > > > >
> > > > > Thanks for KIP-225. Your proposal looks reasonable.
> > > > >
> > > > > Hi, Jiangjie,
> > > > >
> > > > > Do you think the approach that KIP-225 proposes is better for
> > exposing
> > > > the
> > > > > per partition metric? Also, do we really need the per partition
> > > > > record-lag-avg
> > > > > and record-lag-max? It seems that an external monitoring system can
> > > > always
> > > > > derive that from the per partition record-lag.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> > > charly.molter@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Jun, Hu,
> > > > > >
> > > > > > I have KIP-225 open for adding tags to records-lag:
> > > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > > action?pageId=74686649
> > > > > >
> > > > > > I have a patch more or less ready so I could probably get the fix
> > > > checked
> > > > > > in (after the vote) and you could build on top of it. Otherwise
> we
> > > > could
> > > > > > merge both KIPs if you want but they do sound different to me.
> > > > > >
> > > > > > Thanks!
> > > > > > Charly
> > > > > >
> > > > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > >
> > > > > > > Let me double confirm with your comments:
> > > > > > >
> > > > > > > 1 remove partition-level records-lead-avg and records-lead-min
> > > since
> > > > > they
> > > > > > > both can be deduced by external monitoring system.
> > > > > > >
> > > > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > > > >
> > > > > > >
> > > > > > > If they are the case you expect, do we need to do the same
> thing
> > > for
> > > > > > those
> > > > > > > `lag` metrics? Seems partition-level records-lag metrics are
> not
> > > > tagged
> > > > > > > with topic&partition information which might deserve a bug.
> > > > > > >
> > > > > > >
> > > > > > > huxihx
> > > > > > >
> > > > > > >
> > > > > > > ________________________________
> > > > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > > > 发送时间: 2017年11月14日 12:44
> > > > > > > 收件人: dev@kafka.apache.org
> > > > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > > per-partition
> > > > > > > lead metrics to KafkaConsumer
> > > > > > >
> > > > > > > Hi, Hu,
> > > > > > >
> > > > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > > id="{client-id}".
> > > > > > > So, it probably makes sense for records-lead-min to be an
> > attribute
> > > > > under
> > > > > > > the same mbean.
> > > > > > >
> > > > > > > The partition level records-lead can probably be an attribute
> for
> > > the
> > > > > > mbean
> > > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > > > where topic and partition are the tags. This matches the topic
> > > level
> > > > > > mbeans
> > > > > > > that we have in the consumer. I am not sure what the per
> > partition
> > > > > level
> > > > > > > records-lead-min and records-lead-avg are. Are they the min/avg
> > of
> > > > the
> > > > > > lead
> > > > > > > since the consumer is started? I am not sure we need those
> since
> > an
> > > > > > > external monitoring system can always derive them from
> > > records-lead.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Jun,
> > > > > > > >
> > > > > > > > Thanks for the feedback. Some things need to make sure.
> > > Currently,
> > > > > > these
> > > > > > > > new-added metrics follow the exact naming convention with
> those
> > > > 'lag'
> > > > > > > > counterparts, as shown below:
> > > > > > > >
> > > > > > > >
> > > > > > > > Consumer-level metric:
> > > > > > > >
> > > > > > > > records-lag-max ==> records-lead-min
> > > > > > > >
> > > > > > > >
> > > > > > > > Partition-level metrics:
> > > > > > > >
> > > > > > > > <topic>-<partitionId>.records-lag ==>
> > > > > <topic>-<partitionId>.
> > > > > > > > records-lead
> > > > > > > >
> > > > > > > > <topic>-<partitionId>.records-lag-max ==>
> > <topic>-<partitionId>.
> > > > > > > > records-lead-min
> > > > > > > >
> > > > > > > > <topic>-<partitionId>.records-lag-avg ==>
> > > <topic>-<partitionId>.
> > > > > > > > records-lead-avg
> > > > > > > >
> > > > > > > >
> > > > > > > > Correct me if I am wrong, but what you mentioned
> > > `*records-lead-avg
> > > > > and
> > > > > > > > records-lead-min don't need the partition prefix since they
> are
> > > > > > > aggregates
> > > > > > > > across all partitions*` seemed to break the naming rule
> above.
> > Do
> > > > we
> > > > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > > > >
> > > > > > > >
> > > > > > > > huxihx
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ------------------------------
> > > > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > > > *收件人:* dev@kafka.apache.org
> > > > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > > per-partition
> > > > > > > > lead metrics to KafkaConsumer
> > > > > > > >
> > > > > > > > Hi, Hu,
> > > > > > > >
> > > > > > > > Thanks for the KIP. Looks good overall. Could you document
> the
> > > > mbean
> > > > > > name
> > > > > > > > for the new metrics? We probably want the name to be
> consistent
> > > > with
> > > > > > > > records-max-lag as described in
> > > > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http://
> > kafka.apache.org/documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > seems
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http://
> > kafka.apache.org/documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > that
> > > > > > > [http://apache-kafka.org/images/apache-kafka.png
> > >
> > > [http://apache-kafka.org/images/apache-kafka.png]
> > >
> > > ]<http:
> > > > > > //kafka.apache.org/
> > > > > > > documentation/#monitoring>
> > > > > > >
> > > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring
> >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > <http://kafka.apache.org/documentation/#monitoring>
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > > > [http://apache-kafka.org/images/apache-kafka.png
> > >
> > > [http://apache-kafka.org/images/apache-kafka.png]
> > >
> > > ]<http:
> > > > > > //kafka.apache.org/
> > > > > > > documentation/#monitoring>
> > > > > > >
> > > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring
> >
> > > [http://apache-kafka.org/images/apache-kafka.png]<http://
> > kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > Apache Kafka <http://kafka.apache.org/docum
> > entation/#monitoring>
> > > [http://apache-kafka.org/images/apache-kafka.png]<http://
> > kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > > > [http://apache-kafka.org/images/apache-kafka.png
> > >
> > > [http://apache-kafka.org/images/apache-kafka.png]
> > >
> > > ]<http:
> > > > > > //kafka.apache.org/
> > > > > > > documentation/#monitoring>
> > > > > > >
> > > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring
> >
> > > [http://apache-kafka.org/images/apache-kafka.png]<http://
> > kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > kafka.apache.org
> > > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> > use
> > > > > cases
> > > > > > > for
> > > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > > action,
> > > > > > see
> > > > > > > > this blog post.
> > > > > > > >
> > > > > > > >
> > > > > > > > records-lead-avg and records-lead-min don't need the
> partition
> > > > prefix
> > > > > > > since
> > > > > > > > they are aggregates across all partitions. For records-lead,
> it
> > > > seems
> > > > > > > that
> > > > > > > > it's better to add the topic partition as a tag, instead of
> as
> > a
> > > > > prefix
> > > > > > > in
> > > > > > > > the metric name.
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > > > https://cwiki.apache.
> > > > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> > > concerning
> > > > > > > adding
> > > > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to
> leave
> > > your
> > > > > > > > comments
> > > > > > > > > here. Thanks in advance.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Charly Molter
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Charly Molter
> > > >
> > >
> >
>
答复: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Hi Rao Jun,
Already updated the patch per your suggestion and it seems there are no further feedbacks on this KIP. Could we vote now?
________________________________
发件人: Jun Rao <ju...@confluent.io>
发送时间: 2017年11月23日 6:21
收件人: dev@kafka.apache.org
主题: Re: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi, Hu,
There are two types of names. One is the sensor name, which has to be
unique. It's fine if the sensor name includes the topic/partition as the
prefix since the sensor name is only a string and is not exposed to jmx.
The second name is the metric name, which will be used in jmx. Currently,
the existing lag metric name uses topic/partition as the prefix. KIP-225
tries to change the metric name to use topic/partition as the tag. We can
just do the same thing for lead by using tags in the metric name.
Thanks,
Jun
On Mon, Nov 20, 2017 at 10:14 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi Jun,
>
>
> Seems the prefix that is used to be the unique Sensor name cannot be
> removed, so should we keep the prefix?
>
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月21日 3:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> For the new partition level metrics that you are adding, it seems it's
> better to just add the topic/partition tag instead of using them in the
> prefix. For the existing lag metrics, we can fix them in KIP-225.
>
> Thanks,
>
> Jun
>
> On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Jun,
> >
> >
> > Thanks for the comments. Do you think it'd better to add topic/partition
> > tags for those metrics as well as keep the prefix? If those prefixes
> should
> > really be removed, does this KIP need to do the same thing for `lag`
> ones?
> >
> > ________________________________
> > 发件人: Jun Rao <ju...@confluent.io>
> > 发送时间: 2017年11月18日 8:55
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition lead metrics to KafkaConsumer
> >
> > Hi, Charly,
> >
> > Thanks for the input. It makes sense.
> >
> > Hi, Hu,
> >
> > Perhaps we can keep the per partition records-lead-min and
> records-lead-avg
> > as you had before, but just add the topic and the partition as the tags
> > instead of prefix of the metric name.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
> > wrote:
> >
> > > Hi Jun, Jiangle,
> > >
> > > I'd just like to clarify that KIP-225 seems to be using per partition
> > > metric the same way as KIP-223 seems to be doing.
> > >
> > > I believe avg and max are still necessary because the MetricsReporter
> > > doesn't work in a "push" manner and the "Value" measurableStat will
> only
> > > keep the last recorded entry.
> > > Therefore a MetricsReporter usually polls to grab a current view with
> > Value
> > > this view is incomplete so it becomes not possible to compute the
> > > Max/Min/Avg.
> > > Max/Min/Avg uses SampledStats which work with a rolling window of
> samples
> > > and therefore periodic polling would work.
> > >
> > > This is why I believe it's necessary to keep Avg, Min and Max for these
> > > metrics as otherwise we wouldn't be able to recompute it in an external
> > > monitoring system.
> > >
> > > Am I wrong thinking this?
> > >
> > > Thanks,
> > > Charly
> > >
> > >
> > > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Charly,
> > > >
> > > > Thanks for KIP-225. Your proposal looks reasonable.
> > > >
> > > > Hi, Jiangjie,
> > > >
> > > > Do you think the approach that KIP-225 proposes is better for
> exposing
> > > the
> > > > per partition metric? Also, do we really need the per partition
> > > > record-lag-avg
> > > > and record-lag-max? It seems that an external monitoring system can
> > > always
> > > > derive that from the per partition record-lag.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> > charly.molter@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Jun, Hu,
> > > > >
> > > > > I have KIP-225 open for adding tags to records-lag:
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=74686649
> > > > >
> > > > > I have a patch more or less ready so I could probably get the fix
> > > checked
> > > > > in (after the vote) and you could build on top of it. Otherwise we
> > > could
> > > > > merge both KIPs if you want but they do sound different to me.
> > > > >
> > > > > Thanks!
> > > > > Charly
> > > > >
> > > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > >
> > > > > > Let me double confirm with your comments:
> > > > > >
> > > > > > 1 remove partition-level records-lead-avg and records-lead-min
> > since
> > > > they
> > > > > > both can be deduced by external monitoring system.
> > > > > >
> > > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > > >
> > > > > >
> > > > > > If they are the case you expect, do we need to do the same thing
> > for
> > > > > those
> > > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > > tagged
> > > > > > with topic&partition information which might deserve a bug.
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > > 发送时间: 2017年11月14日 12:44
> > > > > > 收件人: dev@kafka.apache.org
> > > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}".
> > > > > > So, it probably makes sense for records-lead-min to be an
> attribute
> > > > under
> > > > > > the same mbean.
> > > > > >
> > > > > > The partition level records-lead can probably be an attribute for
> > the
> > > > > mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > > where topic and partition are the tags. This matches the topic
> > level
> > > > > mbeans
> > > > > > that we have in the consumer. I am not sure what the per
> partition
> > > > level
> > > > > > records-lead-min and records-lead-avg are. Are they the min/avg
> of
> > > the
> > > > > lead
> > > > > > since the consumer is started? I am not sure we need those since
> an
> > > > > > external monitoring system can always derive them from
> > records-lead.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > > Thanks for the feedback. Some things need to make sure.
> > Currently,
> > > > > these
> > > > > > > new-added metrics follow the exact naming convention with those
> > > 'lag'
> > > > > > > counterparts, as shown below:
> > > > > > >
> > > > > > >
> > > > > > > Consumer-level metric:
> > > > > > >
> > > > > > > records-lag-max ==> records-lead-min
> > > > > > >
> > > > > > >
> > > > > > > Partition-level metrics:
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag ==>
> > > > <topic>-<partitionId>.
> > > > > > > records-lead
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-max ==>
> <topic>-<partitionId>.
> > > > > > > records-lead-min
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-avg ==>
> > <topic>-<partitionId>.
> > > > > > > records-lead-avg
> > > > > > >
> > > > > > >
> > > > > > > Correct me if I am wrong, but what you mentioned
> > `*records-lead-avg
> > > > and
> > > > > > > records-lead-min don't need the partition prefix since they are
> > > > > > aggregates
> > > > > > > across all partitions*` seemed to break the naming rule above.
> Do
> > > we
> > > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > > >
> > > > > > >
> > > > > > > huxihx
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ------------------------------
> > > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > > *收件人:* dev@kafka.apache.org
> > > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > > lead metrics to KafkaConsumer
> > > > > > >
> > > > > > > Hi, Hu,
> > > > > > >
> > > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > > mbean
> > > > > name
> > > > > > > for the new metrics? We probably want the name to be consistent
> > > with
> > > > > > > records-max-lag as described in
> > > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > seems
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > that
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > <http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Apache Kafka <http://kafka.apache.org/docum
> entation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > > records-lead-avg and records-lead-min don't need the partition
> > > prefix
> > > > > > since
> > > > > > > they are aggregates across all partitions. For records-lead, it
> > > seems
> > > > > > that
> > > > > > > it's better to add the topic partition as a tag, instead of as
> a
> > > > prefix
> > > > > > in
> > > > > > > the metric name.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > >
> > > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > > https://cwiki.apache.
> > > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> > concerning
> > > > > > adding
> > > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave
> > your
> > > > > > > comments
> > > > > > > > here. Thanks in advance.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Charly Molter
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Charly Molter
> > >
> >
>
答复: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Jun,
Sound reasonable. Already submitted a new patch<https://github.com/apache/kafka/pull/4191> and the KIP<https://cwiki.apache.org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+lead+and+per-partition+lead+metrics+to+KafkaConsumer> was also updated. Please review again. Thanks.
________________________________
发件人: Jun Rao <ju...@confluent.io>
发送时间: 2017年11月23日 6:21
收件人: dev@kafka.apache.org
主题: Re: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi, Hu,
There are two types of names. One is the sensor name, which has to be
unique. It's fine if the sensor name includes the topic/partition as the
prefix since the sensor name is only a string and is not exposed to jmx.
The second name is the metric name, which will be used in jmx. Currently,
the existing lag metric name uses topic/partition as the prefix. KIP-225
tries to change the metric name to use topic/partition as the tag. We can
just do the same thing for lead by using tags in the metric name.
Thanks,
Jun
On Mon, Nov 20, 2017 at 10:14 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi Jun,
>
>
> Seems the prefix that is used to be the unique Sensor name cannot be
> removed, so should we keep the prefix?
>
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月21日 3:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> For the new partition level metrics that you are adding, it seems it's
> better to just add the topic/partition tag instead of using them in the
> prefix. For the existing lag metrics, we can fix them in KIP-225.
>
> Thanks,
>
> Jun
>
> On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Jun,
> >
> >
> > Thanks for the comments. Do you think it'd better to add topic/partition
> > tags for those metrics as well as keep the prefix? If those prefixes
> should
> > really be removed, does this KIP need to do the same thing for `lag`
> ones?
> >
> > ________________________________
> > 发件人: Jun Rao <ju...@confluent.io>
> > 发送时间: 2017年11月18日 8:55
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition lead metrics to KafkaConsumer
> >
> > Hi, Charly,
> >
> > Thanks for the input. It makes sense.
> >
> > Hi, Hu,
> >
> > Perhaps we can keep the per partition records-lead-min and
> records-lead-avg
> > as you had before, but just add the topic and the partition as the tags
> > instead of prefix of the metric name.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
> > wrote:
> >
> > > Hi Jun, Jiangle,
> > >
> > > I'd just like to clarify that KIP-225 seems to be using per partition
> > > metric the same way as KIP-223 seems to be doing.
> > >
> > > I believe avg and max are still necessary because the MetricsReporter
> > > doesn't work in a "push" manner and the "Value" measurableStat will
> only
> > > keep the last recorded entry.
> > > Therefore a MetricsReporter usually polls to grab a current view with
> > Value
> > > this view is incomplete so it becomes not possible to compute the
> > > Max/Min/Avg.
> > > Max/Min/Avg uses SampledStats which work with a rolling window of
> samples
> > > and therefore periodic polling would work.
> > >
> > > This is why I believe it's necessary to keep Avg, Min and Max for these
> > > metrics as otherwise we wouldn't be able to recompute it in an external
> > > monitoring system.
> > >
> > > Am I wrong thinking this?
> > >
> > > Thanks,
> > > Charly
> > >
> > >
> > > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Charly,
> > > >
> > > > Thanks for KIP-225. Your proposal looks reasonable.
> > > >
> > > > Hi, Jiangjie,
> > > >
> > > > Do you think the approach that KIP-225 proposes is better for
> exposing
> > > the
> > > > per partition metric? Also, do we really need the per partition
> > > > record-lag-avg
> > > > and record-lag-max? It seems that an external monitoring system can
> > > always
> > > > derive that from the per partition record-lag.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> > charly.molter@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Jun, Hu,
> > > > >
> > > > > I have KIP-225 open for adding tags to records-lag:
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=74686649
> > > > >
> > > > > I have a patch more or less ready so I could probably get the fix
> > > checked
> > > > > in (after the vote) and you could build on top of it. Otherwise we
> > > could
> > > > > merge both KIPs if you want but they do sound different to me.
> > > > >
> > > > > Thanks!
> > > > > Charly
> > > > >
> > > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > >
> > > > > > Let me double confirm with your comments:
> > > > > >
> > > > > > 1 remove partition-level records-lead-avg and records-lead-min
> > since
> > > > they
> > > > > > both can be deduced by external monitoring system.
> > > > > >
> > > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > > >
> > > > > >
> > > > > > If they are the case you expect, do we need to do the same thing
> > for
> > > > > those
> > > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > > tagged
> > > > > > with topic&partition information which might deserve a bug.
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > > 发送时间: 2017年11月14日 12:44
> > > > > > 收件人: dev@kafka.apache.org
> > > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}".
> > > > > > So, it probably makes sense for records-lead-min to be an
> attribute
> > > > under
> > > > > > the same mbean.
> > > > > >
> > > > > > The partition level records-lead can probably be an attribute for
> > the
> > > > > mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > > where topic and partition are the tags. This matches the topic
> > level
> > > > > mbeans
> > > > > > that we have in the consumer. I am not sure what the per
> partition
> > > > level
> > > > > > records-lead-min and records-lead-avg are. Are they the min/avg
> of
> > > the
> > > > > lead
> > > > > > since the consumer is started? I am not sure we need those since
> an
> > > > > > external monitoring system can always derive them from
> > records-lead.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > > Thanks for the feedback. Some things need to make sure.
> > Currently,
> > > > > these
> > > > > > > new-added metrics follow the exact naming convention with those
> > > 'lag'
> > > > > > > counterparts, as shown below:
> > > > > > >
> > > > > > >
> > > > > > > Consumer-level metric:
> > > > > > >
> > > > > > > records-lag-max ==> records-lead-min
> > > > > > >
> > > > > > >
> > > > > > > Partition-level metrics:
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag ==>
> > > > <topic>-<partitionId>.
> > > > > > > records-lead
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-max ==>
> <topic>-<partitionId>.
> > > > > > > records-lead-min
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-avg ==>
> > <topic>-<partitionId>.
> > > > > > > records-lead-avg
> > > > > > >
> > > > > > >
> > > > > > > Correct me if I am wrong, but what you mentioned
> > `*records-lead-avg
> > > > and
> > > > > > > records-lead-min don't need the partition prefix since they are
> > > > > > aggregates
> > > > > > > across all partitions*` seemed to break the naming rule above.
> Do
> > > we
> > > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > > >
> > > > > > >
> > > > > > > huxihx
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ------------------------------
> > > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > > *收件人:* dev@kafka.apache.org
> > > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > > lead metrics to KafkaConsumer
> > > > > > >
> > > > > > > Hi, Hu,
> > > > > > >
> > > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > > mbean
> > > > > name
> > > > > > > for the new metrics? We probably want the name to be consistent
> > > with
> > > > > > > records-max-lag as described in
> > > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > seems
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > that
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > <http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Apache Kafka <http://kafka.apache.org/docum
> entation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > > records-lead-avg and records-lead-min don't need the partition
> > > prefix
> > > > > > since
> > > > > > > they are aggregates across all partitions. For records-lead, it
> > > seems
> > > > > > that
> > > > > > > it's better to add the topic partition as a tag, instead of as
> a
> > > > prefix
> > > > > > in
> > > > > > > the metric name.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > >
> > > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > > https://cwiki.apache.
> > > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> > concerning
> > > > > > adding
> > > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave
> > your
> > > > > > > comments
> > > > > > > > here. Thanks in advance.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Charly Molter
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Charly Molter
> > >
> >
>
Re: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and
per-partition lead metrics to KafkaConsumer
Posted by Jun Rao <ju...@confluent.io>.
Hi, Hu,
There are two types of names. One is the sensor name, which has to be
unique. It's fine if the sensor name includes the topic/partition as the
prefix since the sensor name is only a string and is not exposed to jmx.
The second name is the metric name, which will be used in jmx. Currently,
the existing lag metric name uses topic/partition as the prefix. KIP-225
tries to change the metric name to use topic/partition as the tag. We can
just do the same thing for lead by using tags in the metric name.
Thanks,
Jun
On Mon, Nov 20, 2017 at 10:14 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi Jun,
>
>
> Seems the prefix that is used to be the unique Sensor name cannot be
> removed, so should we keep the prefix?
>
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月21日 3:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> For the new partition level metrics that you are adding, it seems it's
> better to just add the topic/partition tag instead of using them in the
> prefix. For the existing lag metrics, we can fix them in KIP-225.
>
> Thanks,
>
> Jun
>
> On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Jun,
> >
> >
> > Thanks for the comments. Do you think it'd better to add topic/partition
> > tags for those metrics as well as keep the prefix? If those prefixes
> should
> > really be removed, does this KIP need to do the same thing for `lag`
> ones?
> >
> > ________________________________
> > 发件人: Jun Rao <ju...@confluent.io>
> > 发送时间: 2017年11月18日 8:55
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition lead metrics to KafkaConsumer
> >
> > Hi, Charly,
> >
> > Thanks for the input. It makes sense.
> >
> > Hi, Hu,
> >
> > Perhaps we can keep the per partition records-lead-min and
> records-lead-avg
> > as you had before, but just add the topic and the partition as the tags
> > instead of prefix of the metric name.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
> > wrote:
> >
> > > Hi Jun, Jiangle,
> > >
> > > I'd just like to clarify that KIP-225 seems to be using per partition
> > > metric the same way as KIP-223 seems to be doing.
> > >
> > > I believe avg and max are still necessary because the MetricsReporter
> > > doesn't work in a "push" manner and the "Value" measurableStat will
> only
> > > keep the last recorded entry.
> > > Therefore a MetricsReporter usually polls to grab a current view with
> > Value
> > > this view is incomplete so it becomes not possible to compute the
> > > Max/Min/Avg.
> > > Max/Min/Avg uses SampledStats which work with a rolling window of
> samples
> > > and therefore periodic polling would work.
> > >
> > > This is why I believe it's necessary to keep Avg, Min and Max for these
> > > metrics as otherwise we wouldn't be able to recompute it in an external
> > > monitoring system.
> > >
> > > Am I wrong thinking this?
> > >
> > > Thanks,
> > > Charly
> > >
> > >
> > > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Charly,
> > > >
> > > > Thanks for KIP-225. Your proposal looks reasonable.
> > > >
> > > > Hi, Jiangjie,
> > > >
> > > > Do you think the approach that KIP-225 proposes is better for
> exposing
> > > the
> > > > per partition metric? Also, do we really need the per partition
> > > > record-lag-avg
> > > > and record-lag-max? It seems that an external monitoring system can
> > > always
> > > > derive that from the per partition record-lag.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> > charly.molter@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Jun, Hu,
> > > > >
> > > > > I have KIP-225 open for adding tags to records-lag:
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=74686649
> > > > >
> > > > > I have a patch more or less ready so I could probably get the fix
> > > checked
> > > > > in (after the vote) and you could build on top of it. Otherwise we
> > > could
> > > > > merge both KIPs if you want but they do sound different to me.
> > > > >
> > > > > Thanks!
> > > > > Charly
> > > > >
> > > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > >
> > > > > > Let me double confirm with your comments:
> > > > > >
> > > > > > 1 remove partition-level records-lead-avg and records-lead-min
> > since
> > > > they
> > > > > > both can be deduced by external monitoring system.
> > > > > >
> > > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > > >
> > > > > >
> > > > > > If they are the case you expect, do we need to do the same thing
> > for
> > > > > those
> > > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > > tagged
> > > > > > with topic&partition information which might deserve a bug.
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > > 发送时间: 2017年11月14日 12:44
> > > > > > 收件人: dev@kafka.apache.org
> > > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}".
> > > > > > So, it probably makes sense for records-lead-min to be an
> attribute
> > > > under
> > > > > > the same mbean.
> > > > > >
> > > > > > The partition level records-lead can probably be an attribute for
> > the
> > > > > mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > > where topic and partition are the tags. This matches the topic
> > level
> > > > > mbeans
> > > > > > that we have in the consumer. I am not sure what the per
> partition
> > > > level
> > > > > > records-lead-min and records-lead-avg are. Are they the min/avg
> of
> > > the
> > > > > lead
> > > > > > since the consumer is started? I am not sure we need those since
> an
> > > > > > external monitoring system can always derive them from
> > records-lead.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > > Thanks for the feedback. Some things need to make sure.
> > Currently,
> > > > > these
> > > > > > > new-added metrics follow the exact naming convention with those
> > > 'lag'
> > > > > > > counterparts, as shown below:
> > > > > > >
> > > > > > >
> > > > > > > Consumer-level metric:
> > > > > > >
> > > > > > > records-lag-max ==> records-lead-min
> > > > > > >
> > > > > > >
> > > > > > > Partition-level metrics:
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag ==>
> > > > <topic>-<partitionId>.
> > > > > > > records-lead
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-max ==>
> <topic>-<partitionId>.
> > > > > > > records-lead-min
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-avg ==>
> > <topic>-<partitionId>.
> > > > > > > records-lead-avg
> > > > > > >
> > > > > > >
> > > > > > > Correct me if I am wrong, but what you mentioned
> > `*records-lead-avg
> > > > and
> > > > > > > records-lead-min don't need the partition prefix since they are
> > > > > > aggregates
> > > > > > > across all partitions*` seemed to break the naming rule above.
> Do
> > > we
> > > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > > >
> > > > > > >
> > > > > > > huxihx
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ------------------------------
> > > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > > *收件人:* dev@kafka.apache.org
> > > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > > lead metrics to KafkaConsumer
> > > > > > >
> > > > > > > Hi, Hu,
> > > > > > >
> > > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > > mbean
> > > > > name
> > > > > > > for the new metrics? We probably want the name to be consistent
> > > with
> > > > > > > records-max-lag as described in
> > > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
> [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > seems
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > that
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > <http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Apache Kafka <http://kafka.apache.org/docum
> entation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > > records-lead-avg and records-lead-min don't need the partition
> > > prefix
> > > > > > since
> > > > > > > they are aggregates across all partitions. For records-lead, it
> > > seems
> > > > > > that
> > > > > > > it's better to add the topic partition as a tag, instead of as
> a
> > > > prefix
> > > > > > in
> > > > > > > the metric name.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > >
> > > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > > https://cwiki.apache.
> > > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> > concerning
> > > > > > adding
> > > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave
> > your
> > > > > > > comments
> > > > > > > > here. Thanks in advance.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Charly Molter
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Charly Molter
> > >
> >
>
REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Hi Jun,
Seems the prefix that is used to be the unique Sensor name cannot be removed, so should we keep the prefix?
________________________________
发件人: Jun Rao <ju...@confluent.io>
发送时间: 2017年11月21日 3:55
收件人: dev@kafka.apache.org
主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi, Hu,
For the new partition level metrics that you are adding, it seems it's
better to just add the topic/partition tag instead of using them in the
prefix. For the existing lag metrics, we can fix them in KIP-225.
Thanks,
Jun
On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
> Jun,
>
>
> Thanks for the comments. Do you think it'd better to add topic/partition
> tags for those metrics as well as keep the prefix? If those prefixes should
> really be removed, does this KIP need to do the same thing for `lag` ones?
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月18日 8:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Charly,
>
> Thanks for the input. It makes sense.
>
> Hi, Hu,
>
> Perhaps we can keep the per partition records-lead-min and records-lead-avg
> as you had before, but just add the topic and the partition as the tags
> instead of prefix of the metric name.
>
> Thanks,
>
> Jun
>
>
>
> On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
> wrote:
>
> > Hi Jun, Jiangle,
> >
> > I'd just like to clarify that KIP-225 seems to be using per partition
> > metric the same way as KIP-223 seems to be doing.
> >
> > I believe avg and max are still necessary because the MetricsReporter
> > doesn't work in a "push" manner and the "Value" measurableStat will only
> > keep the last recorded entry.
> > Therefore a MetricsReporter usually polls to grab a current view with
> Value
> > this view is incomplete so it becomes not possible to compute the
> > Max/Min/Avg.
> > Max/Min/Avg uses SampledStats which work with a rolling window of samples
> > and therefore periodic polling would work.
> >
> > This is why I believe it's necessary to keep Avg, Min and Max for these
> > metrics as otherwise we wouldn't be able to recompute it in an external
> > monitoring system.
> >
> > Am I wrong thinking this?
> >
> > Thanks,
> > Charly
> >
> >
> > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Charly,
> > >
> > > Thanks for KIP-225. Your proposal looks reasonable.
> > >
> > > Hi, Jiangjie,
> > >
> > > Do you think the approach that KIP-225 proposes is better for exposing
> > the
> > > per partition metric? Also, do we really need the per partition
> > > record-lag-avg
> > > and record-lag-max? It seems that an external monitoring system can
> > always
> > > derive that from the per partition record-lag.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> charly.molter@gmail.com>
> > > wrote:
> > >
> > > > Hi Jun, Hu,
> > > >
> > > > I have KIP-225 open for adding tags to records-lag:
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=74686649
> > > >
> > > > I have a patch more or less ready so I could probably get the fix
> > checked
> > > > in (after the vote) and you could build on top of it. Otherwise we
> > could
> > > > merge both KIPs if you want but they do sound different to me.
> > > >
> > > > Thanks!
> > > > Charly
> > > >
> > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > >
> > > > > Let me double confirm with your comments:
> > > > >
> > > > > 1 remove partition-level records-lead-avg and records-lead-min
> since
> > > they
> > > > > both can be deduced by external monitoring system.
> > > > >
> > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > >
> > > > >
> > > > > If they are the case you expect, do we need to do the same thing
> for
> > > > those
> > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > tagged
> > > > > with topic&partition information which might deserve a bug.
> > > > >
> > > > >
> > > > > huxihx
> > > > >
> > > > >
> > > > > ________________________________
> > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > 发送时间: 2017年11月14日 12:44
> > > > > 收件人: dev@kafka.apache.org
> > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > lead metrics to KafkaConsumer
> > > > >
> > > > > Hi, Hu,
> > > > >
> > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}".
> > > > > So, it probably makes sense for records-lead-min to be an attribute
> > > under
> > > > > the same mbean.
> > > > >
> > > > > The partition level records-lead can probably be an attribute for
> the
> > > > mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > where topic and partition are the tags. This matches the topic
> level
> > > > mbeans
> > > > > that we have in the consumer. I am not sure what the per partition
> > > level
> > > > > records-lead-min and records-lead-avg are. Are they the min/avg of
> > the
> > > > lead
> > > > > since the consumer is started? I am not sure we need those since an
> > > > > external monitoring system can always derive them from
> records-lead.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > Thanks for the feedback. Some things need to make sure.
> Currently,
> > > > these
> > > > > > new-added metrics follow the exact naming convention with those
> > 'lag'
> > > > > > counterparts, as shown below:
> > > > > >
> > > > > >
> > > > > > Consumer-level metric:
> > > > > >
> > > > > > records-lag-max ==> records-lead-min
> > > > > >
> > > > > >
> > > > > > Partition-level metrics:
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag ==>
> > > <topic>-<partitionId>.
> > > > > > records-lead
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > > > > records-lead-min
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag-avg ==>
> <topic>-<partitionId>.
> > > > > > records-lead-avg
> > > > > >
> > > > > >
> > > > > > Correct me if I am wrong, but what you mentioned
> `*records-lead-avg
> > > and
> > > > > > records-lead-min don't need the partition prefix since they are
> > > > > aggregates
> > > > > > across all partitions*` seemed to break the naming rule above. Do
> > we
> > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------
> > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > *收件人:* dev@kafka.apache.org
> > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > mbean
> > > > name
> > > > > > for the new metrics? We probably want the name to be consistent
> > with
> > > > > > records-max-lag as described in
> > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> seems
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > that
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > <http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > > records-lead-avg and records-lead-min don't need the partition
> > prefix
> > > > > since
> > > > > > they are aggregates across all partitions. For records-lead, it
> > seems
> > > > > that
> > > > > > it's better to add the topic partition as a tag, instead of as a
> > > prefix
> > > > > in
> > > > > > the metric name.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > >
> > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > https://cwiki.apache.
> > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> concerning
> > > > > adding
> > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave
> your
> > > > > > comments
> > > > > > > here. Thanks in advance.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Charly Molter
> > > >
> > >
> >
> >
> >
> > --
> > Charly Molter
> >
>
Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Jun Rao <ju...@confluent.io>.
Hi, Hu,
For the new partition level metrics that you are adding, it seems it's
better to just add the topic/partition tag instead of using them in the
prefix. For the existing lag metrics, we can fix them in KIP-225.
Thanks,
Jun
On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
> Jun,
>
>
> Thanks for the comments. Do you think it'd better to add topic/partition
> tags for those metrics as well as keep the prefix? If those prefixes should
> really be removed, does this KIP need to do the same thing for `lag` ones?
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月18日 8:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Charly,
>
> Thanks for the input. It makes sense.
>
> Hi, Hu,
>
> Perhaps we can keep the per partition records-lead-min and records-lead-avg
> as you had before, but just add the topic and the partition as the tags
> instead of prefix of the metric name.
>
> Thanks,
>
> Jun
>
>
>
> On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
> wrote:
>
> > Hi Jun, Jiangle,
> >
> > I'd just like to clarify that KIP-225 seems to be using per partition
> > metric the same way as KIP-223 seems to be doing.
> >
> > I believe avg and max are still necessary because the MetricsReporter
> > doesn't work in a "push" manner and the "Value" measurableStat will only
> > keep the last recorded entry.
> > Therefore a MetricsReporter usually polls to grab a current view with
> Value
> > this view is incomplete so it becomes not possible to compute the
> > Max/Min/Avg.
> > Max/Min/Avg uses SampledStats which work with a rolling window of samples
> > and therefore periodic polling would work.
> >
> > This is why I believe it's necessary to keep Avg, Min and Max for these
> > metrics as otherwise we wouldn't be able to recompute it in an external
> > monitoring system.
> >
> > Am I wrong thinking this?
> >
> > Thanks,
> > Charly
> >
> >
> > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Charly,
> > >
> > > Thanks for KIP-225. Your proposal looks reasonable.
> > >
> > > Hi, Jiangjie,
> > >
> > > Do you think the approach that KIP-225 proposes is better for exposing
> > the
> > > per partition metric? Also, do we really need the per partition
> > > record-lag-avg
> > > and record-lag-max? It seems that an external monitoring system can
> > always
> > > derive that from the per partition record-lag.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> charly.molter@gmail.com>
> > > wrote:
> > >
> > > > Hi Jun, Hu,
> > > >
> > > > I have KIP-225 open for adding tags to records-lag:
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=74686649
> > > >
> > > > I have a patch more or less ready so I could probably get the fix
> > checked
> > > > in (after the vote) and you could build on top of it. Otherwise we
> > could
> > > > merge both KIPs if you want but they do sound different to me.
> > > >
> > > > Thanks!
> > > > Charly
> > > >
> > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > >
> > > > > Let me double confirm with your comments:
> > > > >
> > > > > 1 remove partition-level records-lead-avg and records-lead-min
> since
> > > they
> > > > > both can be deduced by external monitoring system.
> > > > >
> > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > >
> > > > >
> > > > > If they are the case you expect, do we need to do the same thing
> for
> > > > those
> > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > tagged
> > > > > with topic&partition information which might deserve a bug.
> > > > >
> > > > >
> > > > > huxihx
> > > > >
> > > > >
> > > > > ________________________________
> > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > 发送时间: 2017年11月14日 12:44
> > > > > 收件人: dev@kafka.apache.org
> > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > lead metrics to KafkaConsumer
> > > > >
> > > > > Hi, Hu,
> > > > >
> > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}".
> > > > > So, it probably makes sense for records-lead-min to be an attribute
> > > under
> > > > > the same mbean.
> > > > >
> > > > > The partition level records-lead can probably be an attribute for
> the
> > > > mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > where topic and partition are the tags. This matches the topic
> level
> > > > mbeans
> > > > > that we have in the consumer. I am not sure what the per partition
> > > level
> > > > > records-lead-min and records-lead-avg are. Are they the min/avg of
> > the
> > > > lead
> > > > > since the consumer is started? I am not sure we need those since an
> > > > > external monitoring system can always derive them from
> records-lead.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > Thanks for the feedback. Some things need to make sure.
> Currently,
> > > > these
> > > > > > new-added metrics follow the exact naming convention with those
> > 'lag'
> > > > > > counterparts, as shown below:
> > > > > >
> > > > > >
> > > > > > Consumer-level metric:
> > > > > >
> > > > > > records-lag-max ==> records-lead-min
> > > > > >
> > > > > >
> > > > > > Partition-level metrics:
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag ==>
> > > <topic>-<partitionId>.
> > > > > > records-lead
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > > > > records-lead-min
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag-avg ==>
> <topic>-<partitionId>.
> > > > > > records-lead-avg
> > > > > >
> > > > > >
> > > > > > Correct me if I am wrong, but what you mentioned
> `*records-lead-avg
> > > and
> > > > > > records-lead-min don't need the partition prefix since they are
> > > > > aggregates
> > > > > > across all partitions*` seemed to break the naming rule above. Do
> > we
> > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------
> > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > *收件人:* dev@kafka.apache.org
> > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > mbean
> > > > name
> > > > > > for the new metrics? We probably want the name to be consistent
> > with
> > > > > > records-max-lag as described in
> > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
> seems
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > that
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > <http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > > records-lead-avg and records-lead-min don't need the partition
> > prefix
> > > > > since
> > > > > > they are aggregates across all partitions. For records-lead, it
> > seems
> > > > > that
> > > > > > it's better to add the topic partition as a tag, instead of as a
> > > prefix
> > > > > in
> > > > > > the metric name.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > >
> > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > https://cwiki.apache.
> > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> concerning
> > > > > adding
> > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave
> your
> > > > > > comments
> > > > > > > here. Thanks in advance.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Charly Molter
> > > >
> > >
> >
> >
> >
> > --
> > Charly Molter
> >
>
答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Jun,
Thanks for the comments. Do you think it'd better to add topic/partition tags for those metrics as well as keep the prefix? If those prefixes should really be removed, does this KIP need to do the same thing for `lag` ones?
________________________________
发件人: Jun Rao <ju...@confluent.io>
发送时间: 2017年11月18日 8:55
收件人: dev@kafka.apache.org
主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi, Charly,
Thanks for the input. It makes sense.
Hi, Hu,
Perhaps we can keep the per partition records-lead-min and records-lead-avg
as you had before, but just add the topic and the partition as the tags
instead of prefix of the metric name.
Thanks,
Jun
On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
wrote:
> Hi Jun, Jiangle,
>
> I'd just like to clarify that KIP-225 seems to be using per partition
> metric the same way as KIP-223 seems to be doing.
>
> I believe avg and max are still necessary because the MetricsReporter
> doesn't work in a "push" manner and the "Value" measurableStat will only
> keep the last recorded entry.
> Therefore a MetricsReporter usually polls to grab a current view with Value
> this view is incomplete so it becomes not possible to compute the
> Max/Min/Avg.
> Max/Min/Avg uses SampledStats which work with a rolling window of samples
> and therefore periodic polling would work.
>
> This is why I believe it's necessary to keep Avg, Min and Max for these
> metrics as otherwise we wouldn't be able to recompute it in an external
> monitoring system.
>
> Am I wrong thinking this?
>
> Thanks,
> Charly
>
>
> On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Charly,
> >
> > Thanks for KIP-225. Your proposal looks reasonable.
> >
> > Hi, Jiangjie,
> >
> > Do you think the approach that KIP-225 proposes is better for exposing
> the
> > per partition metric? Also, do we really need the per partition
> > record-lag-avg
> > and record-lag-max? It seems that an external monitoring system can
> always
> > derive that from the per partition record-lag.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <ch...@gmail.com>
> > wrote:
> >
> > > Hi Jun, Hu,
> > >
> > > I have KIP-225 open for adding tags to records-lag:
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=74686649
> > >
> > > I have a patch more or less ready so I could probably get the fix
> checked
> > > in (after the vote) and you could build on top of it. Otherwise we
> could
> > > merge both KIPs if you want but they do sound different to me.
> > >
> > > Thanks!
> > > Charly
> > >
> > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
> > >
> > > > Jun,
> > > >
> > > >
> > > > Let me double confirm with your comments:
> > > >
> > > > 1 remove partition-level records-lead-avg and records-lead-min since
> > they
> > > > both can be deduced by external monitoring system.
> > > >
> > > > 2 Tag partition-level records-lead with topic&partition info
> > > >
> > > >
> > > > If they are the case you expect, do we need to do the same thing for
> > > those
> > > > `lag` metrics? Seems partition-level records-lag metrics are not
> tagged
> > > > with topic&partition information which might deserve a bug.
> > > >
> > > >
> > > > huxihx
> > > >
> > > >
> > > > ________________________________
> > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > 发送时间: 2017年11月14日 12:44
> > > > 收件人: dev@kafka.apache.org
> > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition
> > > > lead metrics to KafkaConsumer
> > > >
> > > > Hi, Hu,
> > > >
> > > > Currently, records-lag-max is an attribute for the mbean
> > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > id="{client-id}".
> > > > So, it probably makes sense for records-lead-min to be an attribute
> > under
> > > > the same mbean.
> > > >
> > > > The partition level records-lead can probably be an attribute for the
> > > mbean
> > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > id="{client-id}",topic=topic1,partition=0,
> > > > where topic and partition are the tags. This matches the topic level
> > > mbeans
> > > > that we have in the consumer. I am not sure what the per partition
> > level
> > > > records-lead-min and records-lead-avg are. Are they the min/avg of
> the
> > > lead
> > > > since the consumer is started? I am not sure we need those since an
> > > > external monitoring system can always derive them from records-lead.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Thanks for the feedback. Some things need to make sure. Currently,
> > > these
> > > > > new-added metrics follow the exact naming convention with those
> 'lag'
> > > > > counterparts, as shown below:
> > > > >
> > > > >
> > > > > Consumer-level metric:
> > > > >
> > > > > records-lag-max ==> records-lead-min
> > > > >
> > > > >
> > > > > Partition-level metrics:
> > > > >
> > > > > <topic>-<partitionId>.records-lag ==>
> > <topic>-<partitionId>.
> > > > > records-lead
> > > > >
> > > > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > > > records-lead-min
> > > > >
> > > > > <topic>-<partitionId>.records-lag-avg ==> <topic>-<partitionId>.
> > > > > records-lead-avg
> > > > >
> > > > >
> > > > > Correct me if I am wrong, but what you mentioned `*records-lead-avg
> > and
> > > > > records-lead-min don't need the partition prefix since they are
> > > > aggregates
> > > > > across all partitions*` seemed to break the naming rule above. Do
> we
> > > > > still have to keep the same rule with the "lag" metrics?
> > > > >
> > > > >
> > > > > huxihx
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ------------------------------
> > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > *发送时间:* 2017年11月14日 1:48
> > > > > *收件人:* dev@kafka.apache.org
> > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition
> > > > > lead metrics to KafkaConsumer
> > > > >
> > > > > Hi, Hu,
> > > > >
> > > > > Thanks for the KIP. Looks good overall. Could you document the
> mbean
> > > name
> > > > > for the new metrics? We probably want the name to be consistent
> with
> > > > > records-max-lag as described in
> > > > > http://kafka.apache.org/documentation/#monitoring. Also, it seems
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > that
> > > > [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http:
> > > //kafka.apache.org/
> > > > documentation/#monitoring>
> > > >
> > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > >
> > > > > <http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > > [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http:
> > > //kafka.apache.org/
> > > > documentation/#monitoring>
> > > >
> > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > >
> > > > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > > [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http:
> > > //kafka.apache.org/
> > > > documentation/#monitoring>
> > > >
> > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > >
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > > records-lead-avg and records-lead-min don't need the partition
> prefix
> > > > since
> > > > > they are aggregates across all partitions. For records-lead, it
> seems
> > > > that
> > > > > it's better to add the topic partition as a tag, instead of as a
> > prefix
> > > > in
> > > > > the metric name.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > >
> > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > https://cwiki.apache.
> > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning
> > > > adding
> > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave your
> > > > > comments
> > > > > > here. Thanks in advance.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Charly Molter
> > >
> >
>
>
>
> --
> Charly Molter
>
Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Jun Rao <ju...@confluent.io>.
Hi, Charly,
Thanks for the input. It makes sense.
Hi, Hu,
Perhaps we can keep the per partition records-lead-min and records-lead-avg
as you had before, but just add the topic and the partition as the tags
instead of prefix of the metric name.
Thanks,
Jun
On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
wrote:
> Hi Jun, Jiangle,
>
> I'd just like to clarify that KIP-225 seems to be using per partition
> metric the same way as KIP-223 seems to be doing.
>
> I believe avg and max are still necessary because the MetricsReporter
> doesn't work in a "push" manner and the "Value" measurableStat will only
> keep the last recorded entry.
> Therefore a MetricsReporter usually polls to grab a current view with Value
> this view is incomplete so it becomes not possible to compute the
> Max/Min/Avg.
> Max/Min/Avg uses SampledStats which work with a rolling window of samples
> and therefore periodic polling would work.
>
> This is why I believe it's necessary to keep Avg, Min and Max for these
> metrics as otherwise we wouldn't be able to recompute it in an external
> monitoring system.
>
> Am I wrong thinking this?
>
> Thanks,
> Charly
>
>
> On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Charly,
> >
> > Thanks for KIP-225. Your proposal looks reasonable.
> >
> > Hi, Jiangjie,
> >
> > Do you think the approach that KIP-225 proposes is better for exposing
> the
> > per partition metric? Also, do we really need the per partition
> > record-lag-avg
> > and record-lag-max? It seems that an external monitoring system can
> always
> > derive that from the per partition record-lag.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <ch...@gmail.com>
> > wrote:
> >
> > > Hi Jun, Hu,
> > >
> > > I have KIP-225 open for adding tags to records-lag:
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=74686649
> > >
> > > I have a patch more or less ready so I could probably get the fix
> checked
> > > in (after the vote) and you could build on top of it. Otherwise we
> could
> > > merge both KIPs if you want but they do sound different to me.
> > >
> > > Thanks!
> > > Charly
> > >
> > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
> > >
> > > > Jun,
> > > >
> > > >
> > > > Let me double confirm with your comments:
> > > >
> > > > 1 remove partition-level records-lead-avg and records-lead-min since
> > they
> > > > both can be deduced by external monitoring system.
> > > >
> > > > 2 Tag partition-level records-lead with topic&partition info
> > > >
> > > >
> > > > If they are the case you expect, do we need to do the same thing for
> > > those
> > > > `lag` metrics? Seems partition-level records-lag metrics are not
> tagged
> > > > with topic&partition information which might deserve a bug.
> > > >
> > > >
> > > > huxihx
> > > >
> > > >
> > > > ________________________________
> > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > 发送时间: 2017年11月14日 12:44
> > > > 收件人: dev@kafka.apache.org
> > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition
> > > > lead metrics to KafkaConsumer
> > > >
> > > > Hi, Hu,
> > > >
> > > > Currently, records-lag-max is an attribute for the mbean
> > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > id="{client-id}".
> > > > So, it probably makes sense for records-lead-min to be an attribute
> > under
> > > > the same mbean.
> > > >
> > > > The partition level records-lead can probably be an attribute for the
> > > mbean
> > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > id="{client-id}",topic=topic1,partition=0,
> > > > where topic and partition are the tags. This matches the topic level
> > > mbeans
> > > > that we have in the consumer. I am not sure what the per partition
> > level
> > > > records-lead-min and records-lead-avg are. Are they the min/avg of
> the
> > > lead
> > > > since the consumer is started? I am not sure we need those since an
> > > > external monitoring system can always derive them from records-lead.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Thanks for the feedback. Some things need to make sure. Currently,
> > > these
> > > > > new-added metrics follow the exact naming convention with those
> 'lag'
> > > > > counterparts, as shown below:
> > > > >
> > > > >
> > > > > Consumer-level metric:
> > > > >
> > > > > records-lag-max ==> records-lead-min
> > > > >
> > > > >
> > > > > Partition-level metrics:
> > > > >
> > > > > <topic>-<partitionId>.records-lag ==>
> > <topic>-<partitionId>.
> > > > > records-lead
> > > > >
> > > > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > > > records-lead-min
> > > > >
> > > > > <topic>-<partitionId>.records-lag-avg ==> <topic>-<partitionId>.
> > > > > records-lead-avg
> > > > >
> > > > >
> > > > > Correct me if I am wrong, but what you mentioned `*records-lead-avg
> > and
> > > > > records-lead-min don't need the partition prefix since they are
> > > > aggregates
> > > > > across all partitions*` seemed to break the naming rule above. Do
> we
> > > > > still have to keep the same rule with the "lag" metrics?
> > > > >
> > > > >
> > > > > huxihx
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ------------------------------
> > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > *发送时间:* 2017年11月14日 1:48
> > > > > *收件人:* dev@kafka.apache.org
> > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition
> > > > > lead metrics to KafkaConsumer
> > > > >
> > > > > Hi, Hu,
> > > > >
> > > > > Thanks for the KIP. Looks good overall. Could you document the
> mbean
> > > name
> > > > > for the new metrics? We probably want the name to be consistent
> with
> > > > > records-max-lag as described in
> > > > > http://kafka.apache.org/documentation/#monitoring. Also, it seems
> > that
> > > > [http://apache-kafka.org/images/apache-kafka.png]<http:
> > > //kafka.apache.org/
> > > > documentation/#monitoring>
> > > >
> > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > >
> > > > > <http://kafka.apache.org/documentation/#monitoring>
> > > > [http://apache-kafka.org/images/apache-kafka.png]<http:
> > > //kafka.apache.org/
> > > > documentation/#monitoring>
> > > >
> > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > >
> > > > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
> > > > [http://apache-kafka.org/images/apache-kafka.png]<http:
> > > //kafka.apache.org/
> > > > documentation/#monitoring>
> > > >
> > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > >
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > > records-lead-avg and records-lead-min don't need the partition
> prefix
> > > > since
> > > > > they are aggregates across all partitions. For records-lead, it
> seems
> > > > that
> > > > > it's better to add the topic partition as a tag, instead of as a
> > prefix
> > > > in
> > > > > the metric name.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > >
> > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > https://cwiki.apache.
> > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning
> > > > adding
> > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave your
> > > > > comments
> > > > > > here. Thanks in advance.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Charly Molter
> > >
> >
>
>
>
> --
> Charly Molter
>
Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by charly molter <ch...@gmail.com>.
Hi Jun, Jiangle,
I'd just like to clarify that KIP-225 seems to be using per partition
metric the same way as KIP-223 seems to be doing.
I believe avg and max are still necessary because the MetricsReporter
doesn't work in a "push" manner and the "Value" measurableStat will only
keep the last recorded entry.
Therefore a MetricsReporter usually polls to grab a current view with Value
this view is incomplete so it becomes not possible to compute the
Max/Min/Avg.
Max/Min/Avg uses SampledStats which work with a rolling window of samples
and therefore periodic polling would work.
This is why I believe it's necessary to keep Avg, Min and Max for these
metrics as otherwise we wouldn't be able to recompute it in an external
monitoring system.
Am I wrong thinking this?
Thanks,
Charly
On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> Hi, Charly,
>
> Thanks for KIP-225. Your proposal looks reasonable.
>
> Hi, Jiangjie,
>
> Do you think the approach that KIP-225 proposes is better for exposing the
> per partition metric? Also, do we really need the per partition
> record-lag-avg
> and record-lag-max? It seems that an external monitoring system can always
> derive that from the per partition record-lag.
>
> Thanks,
>
> Jun
>
> On Tue, Nov 14, 2017 at 6:57 AM, charly molter <ch...@gmail.com>
> wrote:
>
> > Hi Jun, Hu,
> >
> > I have KIP-225 open for adding tags to records-lag:
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=74686649
> >
> > I have a patch more or less ready so I could probably get the fix checked
> > in (after the vote) and you could build on top of it. Otherwise we could
> > merge both KIPs if you want but they do sound different to me.
> >
> > Thanks!
> > Charly
> >
> > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
> >
> > > Jun,
> > >
> > >
> > > Let me double confirm with your comments:
> > >
> > > 1 remove partition-level records-lead-avg and records-lead-min since
> they
> > > both can be deduced by external monitoring system.
> > >
> > > 2 Tag partition-level records-lead with topic&partition info
> > >
> > >
> > > If they are the case you expect, do we need to do the same thing for
> > those
> > > `lag` metrics? Seems partition-level records-lag metrics are not tagged
> > > with topic&partition information which might deserve a bug.
> > >
> > >
> > > huxihx
> > >
> > >
> > > ________________________________
> > > 发件人: Jun Rao <ju...@confluent.io>
> > > 发送时间: 2017年11月14日 12:44
> > > 收件人: dev@kafka.apache.org
> > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> > > lead metrics to KafkaConsumer
> > >
> > > Hi, Hu,
> > >
> > > Currently, records-lag-max is an attribute for the mbean
> > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > id="{client-id}".
> > > So, it probably makes sense for records-lead-min to be an attribute
> under
> > > the same mbean.
> > >
> > > The partition level records-lead can probably be an attribute for the
> > mbean
> > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > id="{client-id}",topic=topic1,partition=0,
> > > where topic and partition are the tags. This matches the topic level
> > mbeans
> > > that we have in the consumer. I am not sure what the per partition
> level
> > > records-lead-min and records-lead-avg are. Are they the min/avg of the
> > lead
> > > since the consumer is started? I am not sure we need those since an
> > > external monitoring system can always derive them from records-lead.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > >
> > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com> wrote:
> > >
> > > > Jun,
> > > >
> > > > Thanks for the feedback. Some things need to make sure. Currently,
> > these
> > > > new-added metrics follow the exact naming convention with those 'lag'
> > > > counterparts, as shown below:
> > > >
> > > >
> > > > Consumer-level metric:
> > > >
> > > > records-lag-max ==> records-lead-min
> > > >
> > > >
> > > > Partition-level metrics:
> > > >
> > > > <topic>-<partitionId>.records-lag ==>
> <topic>-<partitionId>.
> > > > records-lead
> > > >
> > > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > > records-lead-min
> > > >
> > > > <topic>-<partitionId>.records-lag-avg ==> <topic>-<partitionId>.
> > > > records-lead-avg
> > > >
> > > >
> > > > Correct me if I am wrong, but what you mentioned `*records-lead-avg
> and
> > > > records-lead-min don't need the partition prefix since they are
> > > aggregates
> > > > across all partitions*` seemed to break the naming rule above. Do we
> > > > still have to keep the same rule with the "lag" metrics?
> > > >
> > > >
> > > > huxihx
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------
> > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > *发送时间:* 2017年11月14日 1:48
> > > > *收件人:* dev@kafka.apache.org
> > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> > > > lead metrics to KafkaConsumer
> > > >
> > > > Hi, Hu,
> > > >
> > > > Thanks for the KIP. Looks good overall. Could you document the mbean
> > name
> > > > for the new metrics? We probably want the name to be consistent with
> > > > records-max-lag as described in
> > > > http://kafka.apache.org/documentation/#monitoring. Also, it seems
> that
> > > [http://apache-kafka.org/images/apache-kafka.png]<http:
> > //kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > <http://kafka.apache.org/documentation/#monitoring>
> > > [http://apache-kafka.org/images/apache-kafka.png]<http:
> > //kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
> > > [http://apache-kafka.org/images/apache-kafka.png]<http:
> > //kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > > records-lead-avg and records-lead-min don't need the partition prefix
> > > since
> > > > they are aggregates across all partitions. For records-lead, it seems
> > > that
> > > > it's better to add the topic partition as a tag, instead of as a
> prefix
> > > in
> > > > the metric name.
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > https://cwiki.apache.
> > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning
> > > adding
> > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave your
> > > > comments
> > > > > here. Thanks in advance.
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Charly Molter
> >
>
--
Charly Molter
Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Jun Rao <ju...@confluent.io>.
Hi, Charly,
Thanks for KIP-225. Your proposal looks reasonable.
Hi, Jiangjie,
Do you think the approach that KIP-225 proposes is better for exposing the
per partition metric? Also, do we really need the per partition record-lag-avg
and record-lag-max? It seems that an external monitoring system can always
derive that from the per partition record-lag.
Thanks,
Jun
On Tue, Nov 14, 2017 at 6:57 AM, charly molter <ch...@gmail.com>
wrote:
> Hi Jun, Hu,
>
> I have KIP-225 open for adding tags to records-lag:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74686649
>
> I have a patch more or less ready so I could probably get the fix checked
> in (after the vote) and you could build on top of it. Otherwise we could
> merge both KIPs if you want but they do sound different to me.
>
> Thanks!
> Charly
>
> On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Jun,
> >
> >
> > Let me double confirm with your comments:
> >
> > 1 remove partition-level records-lead-avg and records-lead-min since they
> > both can be deduced by external monitoring system.
> >
> > 2 Tag partition-level records-lead with topic&partition info
> >
> >
> > If they are the case you expect, do we need to do the same thing for
> those
> > `lag` metrics? Seems partition-level records-lag metrics are not tagged
> > with topic&partition information which might deserve a bug.
> >
> >
> > huxihx
> >
> >
> > ________________________________
> > 发件人: Jun Rao <ju...@confluent.io>
> > 发送时间: 2017年11月14日 12:44
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> > lead metrics to KafkaConsumer
> >
> > Hi, Hu,
> >
> > Currently, records-lag-max is an attribute for the mbean
> > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > id="{client-id}".
> > So, it probably makes sense for records-lead-min to be an attribute under
> > the same mbean.
> >
> > The partition level records-lead can probably be an attribute for the
> mbean
> > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > id="{client-id}",topic=topic1,partition=0,
> > where topic and partition are the tags. This matches the topic level
> mbeans
> > that we have in the consumer. I am not sure what the per partition level
> > records-lead-min and records-lead-avg are. Are they the min/avg of the
> lead
> > since the consumer is started? I am not sure we need those since an
> > external monitoring system can always derive them from records-lead.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> >
> > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com> wrote:
> >
> > > Jun,
> > >
> > > Thanks for the feedback. Some things need to make sure. Currently,
> these
> > > new-added metrics follow the exact naming convention with those 'lag'
> > > counterparts, as shown below:
> > >
> > >
> > > Consumer-level metric:
> > >
> > > records-lag-max ==> records-lead-min
> > >
> > >
> > > Partition-level metrics:
> > >
> > > <topic>-<partitionId>.records-lag ==> <topic>-<partitionId>.
> > > records-lead
> > >
> > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > records-lead-min
> > >
> > > <topic>-<partitionId>.records-lag-avg ==> <topic>-<partitionId>.
> > > records-lead-avg
> > >
> > >
> > > Correct me if I am wrong, but what you mentioned `*records-lead-avg and
> > > records-lead-min don't need the partition prefix since they are
> > aggregates
> > > across all partitions*` seemed to break the naming rule above. Do we
> > > still have to keep the same rule with the "lag" metrics?
> > >
> > >
> > > huxihx
> > >
> > >
> > >
> > >
> > >
> > > ------------------------------
> > > *发件人:* Jun Rao <ju...@confluent.io>
> > > *发送时间:* 2017年11月14日 1:48
> > > *收件人:* dev@kafka.apache.org
> > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> > > lead metrics to KafkaConsumer
> > >
> > > Hi, Hu,
> > >
> > > Thanks for the KIP. Looks good overall. Could you document the mbean
> name
> > > for the new metrics? We probably want the name to be consistent with
> > > records-max-lag as described in
> > > http://kafka.apache.org/documentation/#monitoring. Also, it seems that
> > [http://apache-kafka.org/images/apache-kafka.png]<http:
> //kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > <http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http:
> //kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http:
> //kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > > records-lead-avg and records-lead-min don't need the partition prefix
> > since
> > > they are aggregates across all partitions. For records-lead, it seems
> > that
> > > it's better to add the topic partition as a tag, instead of as a prefix
> > in
> > > the metric name.
> > >
> > > Jun
> > >
> > >
> > >
> > >
> > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > >
> > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > https://cwiki.apache.
> > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning
> > adding
> > > > new kinds of lag metrics for KafkaConsumer. Be free to leave your
> > > comments
> > > > here. Thanks in advance.
> > > >
> > > >
> > >
> >
>
>
>
> --
> Charly Molter
>
Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by charly molter <ch...@gmail.com>.
Hi Jun, Hu,
I have KIP-225 open for adding tags to records-lag:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74686649
I have a patch more or less ready so I could probably get the fix checked
in (after the vote) and you could build on top of it. Otherwise we could
merge both KIPs if you want but they do sound different to me.
Thanks!
Charly
On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
> Jun,
>
>
> Let me double confirm with your comments:
>
> 1 remove partition-level records-lead-avg and records-lead-min since they
> both can be deduced by external monitoring system.
>
> 2 Tag partition-level records-lead with topic&partition info
>
>
> If they are the case you expect, do we need to do the same thing for those
> `lag` metrics? Seems partition-level records-lag metrics are not tagged
> with topic&partition information which might deserve a bug.
>
>
> huxihx
>
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月14日 12:44
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> Currently, records-lag-max is an attribute for the mbean
> kafka.consumer:type=consumer-fetch-manager-metrics,client-
> id="{client-id}".
> So, it probably makes sense for records-lead-min to be an attribute under
> the same mbean.
>
> The partition level records-lead can probably be an attribute for the mbean
> kafka.consumer:type=consumer-fetch-manager-metrics,client-
> id="{client-id}",topic=topic1,partition=0,
> where topic and partition are the tags. This matches the topic level mbeans
> that we have in the consumer. I am not sure what the per partition level
> records-lead-min and records-lead-avg are. Are they the min/avg of the lead
> since the consumer is started? I am not sure we need those since an
> external monitoring system can always derive them from records-lead.
>
> Thanks,
>
> Jun
>
>
>
>
> On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Jun,
> >
> > Thanks for the feedback. Some things need to make sure. Currently, these
> > new-added metrics follow the exact naming convention with those 'lag'
> > counterparts, as shown below:
> >
> >
> > Consumer-level metric:
> >
> > records-lag-max ==> records-lead-min
> >
> >
> > Partition-level metrics:
> >
> > <topic>-<partitionId>.records-lag ==> <topic>-<partitionId>.
> > records-lead
> >
> > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > records-lead-min
> >
> > <topic>-<partitionId>.records-lag-avg ==> <topic>-<partitionId>.
> > records-lead-avg
> >
> >
> > Correct me if I am wrong, but what you mentioned `*records-lead-avg and
> > records-lead-min don't need the partition prefix since they are
> aggregates
> > across all partitions*` seemed to break the naming rule above. Do we
> > still have to keep the same rule with the "lag" metrics?
> >
> >
> > huxihx
> >
> >
> >
> >
> >
> > ------------------------------
> > *发件人:* Jun Rao <ju...@confluent.io>
> > *发送时间:* 2017年11月14日 1:48
> > *收件人:* dev@kafka.apache.org
> > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> > lead metrics to KafkaConsumer
> >
> > Hi, Hu,
> >
> > Thanks for the KIP. Looks good overall. Could you document the mbean name
> > for the new metrics? We probably want the name to be consistent with
> > records-max-lag as described in
> > http://kafka.apache.org/documentation/#monitoring. Also, it seems that
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > <http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> > records-lead-avg and records-lead-min don't need the partition prefix
> since
> > they are aggregates across all partitions. For records-lead, it seems
> that
> > it's better to add the topic partition as a tag, instead of as a prefix
> in
> > the metric name.
> >
> > Jun
> >
> >
> >
> >
> > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> >
> > > Hi all,
> > >
> > >
> > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > https://cwiki.apache.
> > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning
> adding
> > > new kinds of lag metrics for KafkaConsumer. Be free to leave your
> > comments
> > > here. Thanks in advance.
> > >
> > >
> >
>
--
Charly Molter
答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Jun,
Let me double confirm with your comments:
1 remove partition-level records-lead-avg and records-lead-min since they both can be deduced by external monitoring system.
2 Tag partition-level records-lead with topic&partition info
If they are the case you expect, do we need to do the same thing for those `lag` metrics? Seems partition-level records-lag metrics are not tagged with topic&partition information which might deserve a bug.
huxihx
________________________________
发件人: Jun Rao <ju...@confluent.io>
发送时间: 2017年11月14日 12:44
收件人: dev@kafka.apache.org
主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi, Hu,
Currently, records-lag-max is an attribute for the mbean
kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{client-id}".
So, it probably makes sense for records-lead-min to be an attribute under
the same mbean.
The partition level records-lead can probably be an attribute for the mbean
kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{client-id}",topic=topic1,partition=0,
where topic and partition are the tags. This matches the topic level mbeans
that we have in the consumer. I am not sure what the per partition level
records-lead-min and records-lead-avg are. Are they the min/avg of the lead
since the consumer is started? I am not sure we need those since an
external monitoring system can always derive them from records-lead.
Thanks,
Jun
On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com> wrote:
> Jun,
>
> Thanks for the feedback. Some things need to make sure. Currently, these
> new-added metrics follow the exact naming convention with those 'lag'
> counterparts, as shown below:
>
>
> Consumer-level metric:
>
> records-lag-max ==> records-lead-min
>
>
> Partition-level metrics:
>
> <topic>-<partitionId>.records-lag ==> <topic>-<partitionId>.
> records-lead
>
> <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> records-lead-min
>
> <topic>-<partitionId>.records-lag-avg ==> <topic>-<partitionId>.
> records-lead-avg
>
>
> Correct me if I am wrong, but what you mentioned `*records-lead-avg and
> records-lead-min don't need the partition prefix since they are aggregates
> across all partitions*` seemed to break the naming rule above. Do we
> still have to keep the same rule with the "lag" metrics?
>
>
> huxihx
>
>
>
>
>
> ------------------------------
> *发件人:* Jun Rao <ju...@confluent.io>
> *发送时间:* 2017年11月14日 1:48
> *收件人:* dev@kafka.apache.org
> *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> Thanks for the KIP. Looks good overall. Could you document the mbean name
> for the new metrics? We probably want the name to be consistent with
> records-max-lag as described in
> http://kafka.apache.org/documentation/#monitoring. Also, it seems that
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> <http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
> records-lead-avg and records-lead-min don't need the partition prefix since
> they are aggregates across all partitions. For records-lead, it seems that
> it's better to add the topic partition as a tag, instead of as a prefix in
> the metric name.
>
> Jun
>
>
>
>
> On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Hi all,
> >
> >
> > As per Jun Rao's suggestion, I opened up the KIP-223(
> https://cwiki.apache.
> > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning adding
> > new kinds of lag metrics for KafkaConsumer. Be free to leave your
> comments
> > here. Thanks in advance.
> >
> >
>
Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Jun Rao <ju...@confluent.io>.
Hi, Hu,
Currently, records-lag-max is an attribute for the mbean
kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{client-id}".
So, it probably makes sense for records-lead-min to be an attribute under
the same mbean.
The partition level records-lead can probably be an attribute for the mbean
kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{client-id}",topic=topic1,partition=0,
where topic and partition are the tags. This matches the topic level mbeans
that we have in the consumer. I am not sure what the per partition level
records-lead-min and records-lead-avg are. Are they the min/avg of the lead
since the consumer is started? I am not sure we need those since an
external monitoring system can always derive them from records-lead.
Thanks,
Jun
On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com> wrote:
> Jun,
>
> Thanks for the feedback. Some things need to make sure. Currently, these
> new-added metrics follow the exact naming convention with those 'lag'
> counterparts, as shown below:
>
>
> Consumer-level metric:
>
> records-lag-max ==> records-lead-min
>
>
> Partition-level metrics:
>
> <topic>-<partitionId>.records-lag ==> <topic>-<partitionId>.
> records-lead
>
> <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> records-lead-min
>
> <topic>-<partitionId>.records-lag-avg ==> <topic>-<partitionId>.
> records-lead-avg
>
>
> Correct me if I am wrong, but what you mentioned `*records-lead-avg and
> records-lead-min don't need the partition prefix since they are aggregates
> across all partitions*` seemed to break the naming rule above. Do we
> still have to keep the same rule with the "lag" metrics?
>
>
> huxihx
>
>
>
>
>
> ------------------------------
> *发件人:* Jun Rao <ju...@confluent.io>
> *发送时间:* 2017年11月14日 1:48
> *收件人:* dev@kafka.apache.org
> *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> Thanks for the KIP. Looks good overall. Could you document the mbean name
> for the new metrics? We probably want the name to be consistent with
> records-max-lag as described in
> http://kafka.apache.org/documentation/#monitoring. Also, it seems that
> <http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
> records-lead-avg and records-lead-min don't need the partition prefix since
> they are aggregates across all partitions. For records-lead, it seems that
> it's better to add the topic partition as a tag, instead of as a prefix in
> the metric name.
>
> Jun
>
>
>
>
> On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Hi all,
> >
> >
> > As per Jun Rao's suggestion, I opened up the KIP-223(
> https://cwiki.apache.
> > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning adding
> > new kinds of lag metrics for KafkaConsumer. Be free to leave your
> comments
> > here. Thanks in advance.
> >
> >
>
答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Jun,
Thanks for the feedback. Some things need to make sure. Currently, these new-added metrics follow the exact naming convention with those 'lag' counterparts, as shown below:
[cid:fcb78ea1-d9ac-48f4-a4ac-4fca7208af11]
Consumer-level metric:
records-lag-max ==> records-lead-min
Partition-level metrics:
<topic>-<partitionId>.records-lag ==> <topic>-<partitionId>.records-lead
<topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.records-lead-min
<topic>-<partitionId>.records-lag-avg ==> <topic>-<partitionId>.records-lead-avg
Correct me if I am wrong, but what you mentioned `records-lead-avg and records-lead-min don't need the partition prefix since they are aggregates across all partitions` seemed to break the naming rule above. Do we still have to keep the same rule with the "lag" metrics?
huxihx
________________________________
发件人: Jun Rao <ju...@confluent.io>
发送时间: 2017年11月14日 1:48
收件人: dev@kafka.apache.org
主题: Re: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi, Hu,
Thanks for the KIP. Looks good overall. Could you document the mbean name
for the new metrics? We probably want the name to be consistent with
records-max-lag as described in
http://kafka.apache.org/documentation/#monitoring. Also, it seems that
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
records-lead-avg and records-lead-min don't need the partition prefix since
they are aggregates across all partitions. For records-lead, it seems that
it's better to add the topic partition as a tag, instead of as a prefix in
the metric name.
Jun
On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> Hi all,
>
>
> As per Jun Rao's suggestion, I opened up the KIP-223(https://cwiki.apache.
> org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning adding
> new kinds of lag metrics for KafkaConsumer. Be free to leave your comments
> here. Thanks in advance.
>
>
Re: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead
metrics to KafkaConsumer
Posted by Jun Rao <ju...@confluent.io>.
Hi, Hu,
Thanks for the KIP. Looks good overall. Could you document the mbean name
for the new metrics? We probably want the name to be consistent with
records-max-lag as described in
http://kafka.apache.org/documentation/#monitoring. Also, it seems that
records-lead-avg and records-lead-min don't need the partition prefix since
they are aggregates across all partitions. For records-lead, it seems that
it's better to add the topic partition as a tag, instead of as a prefix in
the metric name.
Jun
On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> Hi all,
>
>
> As per Jun Rao's suggestion, I opened up the KIP-223(https://cwiki.apache.
> org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning adding
> new kinds of lag metrics for KafkaConsumer. Be free to leave your comments
> here. Thanks in advance.
>
>
答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Yu, thanks for your comments. Already explicitly showed the jira number and refined the exact metric names.
________________________________
发件人: Ted Yu <yu...@gmail.com>
发送时间: 2017年11月10日 1:51
收件人: dev@kafka.apache.org
主题: Re: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Can you fill out JIRA number ?
"TOPIC-PARTITION_ID.record-lead"
In the code sample, "records-" is used.
Please make them consistent.
On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> Hi all,
>
>
> As per Jun Rao's suggestion, I opened up the KIP-223(https://cwiki.apache.
> org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning adding
> new kinds of lag metrics for KafkaConsumer. Be free to leave your comments
> here. Thanks in advance.
>
>
Re: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead
metrics to KafkaConsumer
Posted by Ted Yu <yu...@gmail.com>.
Can you fill out JIRA number ?
"TOPIC-PARTITION_ID.record-lead"
In the code sample, "records-" is used.
Please make them consistent.
On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> Hi all,
>
>
> As per Jun Rao's suggestion, I opened up the KIP-223(https://cwiki.apache.
> org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning adding
> new kinds of lag metrics for KafkaConsumer. Be free to leave your comments
> here. Thanks in advance.
>
>