You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Hu Xi <hu...@hotmail.com> on 2017/11/20 06:31:55 UTC
答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Jun,
Thanks for the comments. Do you think it'd better to add topic/partition tags for those metrics as well as keep the prefix? If those prefixes should really be removed, does this KIP need to do the same thing for `lag` ones?
________________________________
发件人: Jun Rao <ju...@confluent.io>
发送时间: 2017年11月18日 8:55
收件人: dev@kafka.apache.org
主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi, Charly,
Thanks for the input. It makes sense.
Hi, Hu,
Perhaps we can keep the per partition records-lead-min and records-lead-avg
as you had before, but just add the topic and the partition as the tags
instead of prefix of the metric name.
Thanks,
Jun
On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
wrote:
> Hi Jun, Jiangle,
>
> I'd just like to clarify that KIP-225 seems to be using per partition
> metric the same way as KIP-223 seems to be doing.
>
> I believe avg and max are still necessary because the MetricsReporter
> doesn't work in a "push" manner and the "Value" measurableStat will only
> keep the last recorded entry.
> Therefore a MetricsReporter usually polls to grab a current view with Value
> this view is incomplete so it becomes not possible to compute the
> Max/Min/Avg.
> Max/Min/Avg uses SampledStats which work with a rolling window of samples
> and therefore periodic polling would work.
>
> This is why I believe it's necessary to keep Avg, Min and Max for these
> metrics as otherwise we wouldn't be able to recompute it in an external
> monitoring system.
>
> Am I wrong thinking this?
>
> Thanks,
> Charly
>
>
> On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Charly,
> >
> > Thanks for KIP-225. Your proposal looks reasonable.
> >
> > Hi, Jiangjie,
> >
> > Do you think the approach that KIP-225 proposes is better for exposing
> the
> > per partition metric? Also, do we really need the per partition
> > record-lag-avg
> > and record-lag-max? It seems that an external monitoring system can
> always
> > derive that from the per partition record-lag.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <ch...@gmail.com>
> > wrote:
> >
> > > Hi Jun, Hu,
> > >
> > > I have KIP-225 open for adding tags to records-lag:
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=74686649
> > >
> > > I have a patch more or less ready so I could probably get the fix
> checked
> > > in (after the vote) and you could build on top of it. Otherwise we
> could
> > > merge both KIPs if you want but they do sound different to me.
> > >
> > > Thanks!
> > > Charly
> > >
> > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
> > >
> > > > Jun,
> > > >
> > > >
> > > > Let me double confirm with your comments:
> > > >
> > > > 1 remove partition-level records-lead-avg and records-lead-min since
> > they
> > > > both can be deduced by external monitoring system.
> > > >
> > > > 2 Tag partition-level records-lead with topic&partition info
> > > >
> > > >
> > > > If they are the case you expect, do we need to do the same thing for
> > > those
> > > > `lag` metrics? Seems partition-level records-lag metrics are not
> tagged
> > > > with topic&partition information which might deserve a bug.
> > > >
> > > >
> > > > huxihx
> > > >
> > > >
> > > > ________________________________
> > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > 发送时间: 2017年11月14日 12:44
> > > > 收件人: dev@kafka.apache.org
> > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition
> > > > lead metrics to KafkaConsumer
> > > >
> > > > Hi, Hu,
> > > >
> > > > Currently, records-lag-max is an attribute for the mbean
> > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > id="{client-id}".
> > > > So, it probably makes sense for records-lead-min to be an attribute
> > under
> > > > the same mbean.
> > > >
> > > > The partition level records-lead can probably be an attribute for the
> > > mbean
> > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > id="{client-id}",topic=topic1,partition=0,
> > > > where topic and partition are the tags. This matches the topic level
> > > mbeans
> > > > that we have in the consumer. I am not sure what the per partition
> > level
> > > > records-lead-min and records-lead-avg are. Are they the min/avg of
> the
> > > lead
> > > > since the consumer is started? I am not sure we need those since an
> > > > external monitoring system can always derive them from records-lead.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Thanks for the feedback. Some things need to make sure. Currently,
> > > these
> > > > > new-added metrics follow the exact naming convention with those
> 'lag'
> > > > > counterparts, as shown below:
> > > > >
> > > > >
> > > > > Consumer-level metric:
> > > > >
> > > > > records-lag-max ==> records-lead-min
> > > > >
> > > > >
> > > > > Partition-level metrics:
> > > > >
> > > > > <topic>-<partitionId>.records-lag ==>
> > <topic>-<partitionId>.
> > > > > records-lead
> > > > >
> > > > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > > > records-lead-min
> > > > >
> > > > > <topic>-<partitionId>.records-lag-avg ==> <topic>-<partitionId>.
> > > > > records-lead-avg
> > > > >
> > > > >
> > > > > Correct me if I am wrong, but what you mentioned `*records-lead-avg
> > and
> > > > > records-lead-min don't need the partition prefix since they are
> > > > aggregates
> > > > > across all partitions*` seemed to break the naming rule above. Do
> we
> > > > > still have to keep the same rule with the "lag" metrics?
> > > > >
> > > > >
> > > > > huxihx
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ------------------------------
> > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > *发送时间:* 2017年11月14日 1:48
> > > > > *收件人:* dev@kafka.apache.org
> > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition
> > > > > lead metrics to KafkaConsumer
> > > > >
> > > > > Hi, Hu,
> > > > >
> > > > > Thanks for the KIP. Looks good overall. Could you document the
> mbean
> > > name
> > > > > for the new metrics? We probably want the name to be consistent
> with
> > > > > records-max-lag as described in
> > > > > http://kafka.apache.org/documentation/#monitoring. Also, it seems
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > that
> > > > [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http:
> > > //kafka.apache.org/
> > > > documentation/#monitoring>
> > > >
> > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > >
> > > > > <http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > > [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http:
> > > //kafka.apache.org/
> > > > documentation/#monitoring>
> > > >
> > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > >
> > > > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > > [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http:
> > > //kafka.apache.org/
> > > > documentation/#monitoring>
> > > >
> > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> > > > kafka.apache.org
> > > > 1.2 Use Cases. Here is a description of a few of the popular use
> cases
> > > for
> > > > Apache Kafka®. For an overview of a number of these areas in action,
> > see
> > > > this blog post.
> > > >
> > > >
> > > >
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > > records-lead-avg and records-lead-min don't need the partition
> prefix
> > > > since
> > > > > they are aggregates across all partitions. For records-lead, it
> seems
> > > > that
> > > > > it's better to add the topic partition as a tag, instead of as a
> > prefix
> > > > in
> > > > > the metric name.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > >
> > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > https://cwiki.apache.
> > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer) concerning
> > > > adding
> > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave your
> > > > > comments
> > > > > > here. Thanks in advance.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Charly Molter
> > >
> >
>
>
>
> --
> Charly Molter
>
Re: 答复: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Jun Rao <ju...@confluent.io>.
Hi, Hu,
Yes. Could you start a separate voting thread?
Thanks,
Jun
On Tue, Nov 28, 2017 at 5:02 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi Rao Jun,
>
>
> Already updated the patch per your suggestion and it seems there are no
> further feedbacks on this KIP. Could we vote now?
>
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月23日 6:21
> 收件人: dev@kafka.apache.org
> 主题: Re: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
> lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> There are two types of names. One is the sensor name, which has to be
> unique. It's fine if the sensor name includes the topic/partition as the
> prefix since the sensor name is only a string and is not exposed to jmx.
> The second name is the metric name, which will be used in jmx. Currently,
> the existing lag metric name uses topic/partition as the prefix. KIP-225
> tries to change the metric name to use topic/partition as the tag. We can
> just do the same thing for lead by using tags in the metric name.
>
> Thanks,
>
> Jun
>
> On Mon, Nov 20, 2017 at 10:14 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Hi Jun,
> >
> >
> > Seems the prefix that is used to be the unique Sensor name cannot be
> > removed, so should we keep the prefix?
> >
> >
> > ________________________________
> > 发件人: Jun Rao <ju...@confluent.io>
> > 发送时间: 2017年11月21日 3:55
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition lead metrics to KafkaConsumer
> >
> > Hi, Hu,
> >
> > For the new partition level metrics that you are adding, it seems it's
> > better to just add the topic/partition tag instead of using them in the
> > prefix. For the existing lag metrics, we can fix them in KIP-225.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
> >
> > > Jun,
> > >
> > >
> > > Thanks for the comments. Do you think it'd better to add
> topic/partition
> > > tags for those metrics as well as keep the prefix? If those prefixes
> > should
> > > really be removed, does this KIP need to do the same thing for `lag`
> > ones?
> > >
> > > ________________________________
> > > 发件人: Jun Rao <ju...@confluent.io>
> > > 发送时间: 2017年11月18日 8:55
> > > 收件人: dev@kafka.apache.org
> > > 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition lead metrics to KafkaConsumer
> > >
> > > Hi, Charly,
> > >
> > > Thanks for the input. It makes sense.
> > >
> > > Hi, Hu,
> > >
> > > Perhaps we can keep the per partition records-lead-min and
> > records-lead-avg
> > > as you had before, but just add the topic and the partition as the tags
> > > instead of prefix of the metric name.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Wed, Nov 15, 2017 at 4:58 AM, charly molter <
> charly.molter@gmail.com>
> > > wrote:
> > >
> > > > Hi Jun, Jiangle,
> > > >
> > > > I'd just like to clarify that KIP-225 seems to be using per partition
> > > > metric the same way as KIP-223 seems to be doing.
> > > >
> > > > I believe avg and max are still necessary because the MetricsReporter
> > > > doesn't work in a "push" manner and the "Value" measurableStat will
> > only
> > > > keep the last recorded entry.
> > > > Therefore a MetricsReporter usually polls to grab a current view with
> > > Value
> > > > this view is incomplete so it becomes not possible to compute the
> > > > Max/Min/Avg.
> > > > Max/Min/Avg uses SampledStats which work with a rolling window of
> > samples
> > > > and therefore periodic polling would work.
> > > >
> > > > This is why I believe it's necessary to keep Avg, Min and Max for
> these
> > > > metrics as otherwise we wouldn't be able to recompute it in an
> external
> > > > monitoring system.
> > > >
> > > > Am I wrong thinking this?
> > > >
> > > > Thanks,
> > > > Charly
> > > >
> > > >
> > > > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Hi, Charly,
> > > > >
> > > > > Thanks for KIP-225. Your proposal looks reasonable.
> > > > >
> > > > > Hi, Jiangjie,
> > > > >
> > > > > Do you think the approach that KIP-225 proposes is better for
> > exposing
> > > > the
> > > > > per partition metric? Also, do we really need the per partition
> > > > > record-lag-avg
> > > > > and record-lag-max? It seems that an external monitoring system can
> > > > always
> > > > > derive that from the per partition record-lag.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> > > charly.molter@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Jun, Hu,
> > > > > >
> > > > > > I have KIP-225 open for adding tags to records-lag:
> > > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > > action?pageId=74686649
> > > > > >
> > > > > > I have a patch more or less ready so I could probably get the fix
> > > > checked
> > > > > > in (after the vote) and you could build on top of it. Otherwise
> we
> > > > could
> > > > > > merge both KIPs if you want but they do sound different to me.
> > > > > >
> > > > > > Thanks!
> > > > > > Charly
> > > > > >
> > > > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > >
> > > > > > > Let me double confirm with your comments:
> > > > > > >
> > > > > > > 1 remove partition-level records-lead-avg and records-lead-min
> > > since
> > > > > they
> > > > > > > both can be deduced by external monitoring system.
> > > > > > >
> > > > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > > > >
> > > > > > >
> > > > > > > If they are the case you expect, do we need to do the same
> thing
> > > for
> > > > > > those
> > > > > > > `lag` metrics? Seems partition-level records-lag metrics are
> not
> > > > tagged
> > > > > > > with topic&partition information which might deserve a bug.
> > > > > > >
> > > > > > >
> > > > > > > huxihx
> > > > > > >
> > > > > > >
> > > > > > > ________________________________
> > > > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > > > 发送时间: 2017年11月14日 12:44
> > > > > > > 收件人: dev@kafka.apache.org
> > > > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > > per-partition
> > > > > > > lead metrics to KafkaConsumer
> > > > > > >
> > > > > > > Hi, Hu,
> > > > > > >
> > > > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > > id="{client-id}".
> > > > > > > So, it probably makes sense for records-lead-min to be an
> > attribute
> > > > > under
> > > > > > > the same mbean.
> > > > > > >
> > > > > > > The partition level records-lead can probably be an attribute
> for
> > > the
> > > > > > mbean
> > > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > > > where topic and partition are the tags. This matches the topic
> > > level
> > > > > > mbeans
> > > > > > > that we have in the consumer. I am not sure what the per
> > partition
> > > > > level
> > > > > > > records-lead-min and records-lead-avg are. Are they the min/avg
> > of
> > > > the
> > > > > > lead
> > > > > > > since the consumer is started? I am not sure we need those
> since
> > an
> > > > > > > external monitoring system can always derive them from
> > > records-lead.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Jun,
> > > > > > > >
> > > > > > > > Thanks for the feedback. Some things need to make sure.
> > > Currently,
> > > > > > these
> > > > > > > > new-added metrics follow the exact naming convention with
> those
> > > > 'lag'
> > > > > > > > counterparts, as shown below:
> > > > > > > >
> > > > > > > >
> > > > > > > > Consumer-level metric:
> > > > > > > >
> > > > > > > > records-lag-max ==> records-lead-min
> > > > > > > >
> > > > > > > >
> > > > > > > > Partition-level metrics:
> > > > > > > >
> > > > > > > > <topic>-<partitionId>.records-lag ==>
> > > > > <topic>-<partitionId>.
> > > > > > > > records-lead
> > > > > > > >
> > > > > > > > <topic>-<partitionId>.records-lag-max ==>
> > <topic>-<partitionId>.
> > > > > > > > records-lead-min
> > > > > > > >
> > > > > > > > <topic>-<partitionId>.records-lag-avg ==>
> > > <topic>-<partitionId>.
> > > > > > > > records-lead-avg
> > > > > > > >
> > > > > > > >
> > > > > > > > Correct me if I am wrong, but what you mentioned
> > > `*records-lead-avg
> > > > > and
> > > > > > > > records-lead-min don't need the partition prefix since they
> are
> > > > > > > aggregates
> > > > > > > > across all partitions*` seemed to break the naming rule
> above.
> > Do
> > > > we
> > > > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > > > >
> > > > > > > >
> > > > > > > > huxihx
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ------------------------------
> > > > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > > > *收件人:* dev@kafka.apache.org
> > > > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > > per-partition
> > > > > > > > lead metrics to KafkaConsumer
> > > > > > > >
> > > > > > > > Hi, Hu,
> > > > > > > >
> > > > > > > > Thanks for the KIP. Looks good overall. Could you document
> the
> > > > mbean
> > > > > > name
> > > > > > > > for the new metrics? We probably want the name to be
> consistent
> > > > with
> > > > > > > > records-max-lag as described in
> > > > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http://
> > kafka.apache.org/documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > seems
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http://
> > kafka.apache.org/documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > that
> > > > > > > [http://apache-kafka.org/images/apache-kafka.png
> > >
> > > [http://apache-kafka.org/images/apache-kafka.png]
> > >
> > > ]<http:
> > > > > > //kafka.apache.org/
> > > > > > > documentation/#monitoring>
> > > > > > >
> > > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring
> >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > <http://kafka.apache.org/documentation/#monitoring>
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > > > [http://apache-kafka.org/images/apache-kafka.png
> > >
> > > [http://apache-kafka.org/images/apache-kafka.png]
> > >
> > > ]<http:
> > > > > > //kafka.apache.org/
> > > > > > > documentation/#monitoring>
> > > > > > >
> > > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring
> >
> > > [http://apache-kafka.org/images/apache-kafka.png]<http://
> > kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > Apache Kafka <http://kafka.apache.org/docum
> > entation/#monitoring>
> > > [http://apache-kafka.org/images/apache-kafka.png]<http://
> > kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > > > [http://apache-kafka.org/images/apache-kafka.png
> > >
> > > [http://apache-kafka.org/images/apache-kafka.png]
> > >
> > > ]<http:
> > > > > > //kafka.apache.org/
> > > > > > > documentation/#monitoring>
> > > > > > >
> > > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring
> >
> > > [http://apache-kafka.org/images/apache-kafka.png]<http://
> > kafka.apache.org/
> > > documentation/#monitoring>
> > >
> > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > > kafka.apache.org
> > > 1.2 Use Cases. Here is a description of a few of the popular use cases
> > for
> > > Apache Kafka®. For an overview of a number of these areas in action,
> see
> > > this blog post.
> > >
> > >
> > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > kafka.apache.org
> > > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> > use
> > > > > cases
> > > > > > > for
> > > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > > action,
> > > > > > see
> > > > > > > > this blog post.
> > > > > > > >
> > > > > > > >
> > > > > > > > records-lead-avg and records-lead-min don't need the
> partition
> > > > prefix
> > > > > > > since
> > > > > > > > they are aggregates across all partitions. For records-lead,
> it
> > > > seems
> > > > > > > that
> > > > > > > > it's better to add the topic partition as a tag, instead of
> as
> > a
> > > > > prefix
> > > > > > > in
> > > > > > > > the metric name.
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > > > https://cwiki.apache.
> > > > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> > > concerning
> > > > > > > adding
> > > > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to
> leave
> > > your
> > > > > > > > comments
> > > > > > > > > here. Thanks in advance.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Charly Molter
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Charly Molter
> > > >
> > >
> >
>
答复: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Hi Rao Jun,
Already updated the patch per your suggestion and it seems there are no further feedbacks on this KIP. Could we vote now?
________________________________
发件人: Jun Rao <ju...@confluent.io>
发送时间: 2017年11月23日 6:21
收件人: dev@kafka.apache.org
主题: Re: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi, Hu,
There are two types of names. One is the sensor name, which has to be
unique. It's fine if the sensor name includes the topic/partition as the
prefix since the sensor name is only a string and is not exposed to jmx.
The second name is the metric name, which will be used in jmx. Currently,
the existing lag metric name uses topic/partition as the prefix. KIP-225
tries to change the metric name to use topic/partition as the tag. We can
just do the same thing for lead by using tags in the metric name.
Thanks,
Jun
On Mon, Nov 20, 2017 at 10:14 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi Jun,
>
>
> Seems the prefix that is used to be the unique Sensor name cannot be
> removed, so should we keep the prefix?
>
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月21日 3:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> For the new partition level metrics that you are adding, it seems it's
> better to just add the topic/partition tag instead of using them in the
> prefix. For the existing lag metrics, we can fix them in KIP-225.
>
> Thanks,
>
> Jun
>
> On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Jun,
> >
> >
> > Thanks for the comments. Do you think it'd better to add topic/partition
> > tags for those metrics as well as keep the prefix? If those prefixes
> should
> > really be removed, does this KIP need to do the same thing for `lag`
> ones?
> >
> > ________________________________
> > 发件人: Jun Rao <ju...@confluent.io>
> > 发送时间: 2017年11月18日 8:55
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition lead metrics to KafkaConsumer
> >
> > Hi, Charly,
> >
> > Thanks for the input. It makes sense.
> >
> > Hi, Hu,
> >
> > Perhaps we can keep the per partition records-lead-min and
> records-lead-avg
> > as you had before, but just add the topic and the partition as the tags
> > instead of prefix of the metric name.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
> > wrote:
> >
> > > Hi Jun, Jiangle,
> > >
> > > I'd just like to clarify that KIP-225 seems to be using per partition
> > > metric the same way as KIP-223 seems to be doing.
> > >
> > > I believe avg and max are still necessary because the MetricsReporter
> > > doesn't work in a "push" manner and the "Value" measurableStat will
> only
> > > keep the last recorded entry.
> > > Therefore a MetricsReporter usually polls to grab a current view with
> > Value
> > > this view is incomplete so it becomes not possible to compute the
> > > Max/Min/Avg.
> > > Max/Min/Avg uses SampledStats which work with a rolling window of
> samples
> > > and therefore periodic polling would work.
> > >
> > > This is why I believe it's necessary to keep Avg, Min and Max for these
> > > metrics as otherwise we wouldn't be able to recompute it in an external
> > > monitoring system.
> > >
> > > Am I wrong thinking this?
> > >
> > > Thanks,
> > > Charly
> > >
> > >
> > > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Charly,
> > > >
> > > > Thanks for KIP-225. Your proposal looks reasonable.
> > > >
> > > > Hi, Jiangjie,
> > > >
> > > > Do you think the approach that KIP-225 proposes is better for
> exposing
> > > the
> > > > per partition metric? Also, do we really need the per partition
> > > > record-lag-avg
> > > > and record-lag-max? It seems that an external monitoring system can
> > > always
> > > > derive that from the per partition record-lag.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> > charly.molter@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Jun, Hu,
> > > > >
> > > > > I have KIP-225 open for adding tags to records-lag:
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=74686649
> > > > >
> > > > > I have a patch more or less ready so I could probably get the fix
> > > checked
> > > > > in (after the vote) and you could build on top of it. Otherwise we
> > > could
> > > > > merge both KIPs if you want but they do sound different to me.
> > > > >
> > > > > Thanks!
> > > > > Charly
> > > > >
> > > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > >
> > > > > > Let me double confirm with your comments:
> > > > > >
> > > > > > 1 remove partition-level records-lead-avg and records-lead-min
> > since
> > > > they
> > > > > > both can be deduced by external monitoring system.
> > > > > >
> > > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > > >
> > > > > >
> > > > > > If they are the case you expect, do we need to do the same thing
> > for
> > > > > those
> > > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > > tagged
> > > > > > with topic&partition information which might deserve a bug.
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > > 发送时间: 2017年11月14日 12:44
> > > > > > 收件人: dev@kafka.apache.org
> > > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}".
> > > > > > So, it probably makes sense for records-lead-min to be an
> attribute
> > > > under
> > > > > > the same mbean.
> > > > > >
> > > > > > The partition level records-lead can probably be an attribute for
> > the
> > > > > mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > > where topic and partition are the tags. This matches the topic
> > level
> > > > > mbeans
> > > > > > that we have in the consumer. I am not sure what the per
> partition
> > > > level
> > > > > > records-lead-min and records-lead-avg are. Are they the min/avg
> of
> > > the
> > > > > lead
> > > > > > since the consumer is started? I am not sure we need those since
> an
> > > > > > external monitoring system can always derive them from
> > records-lead.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > > Thanks for the feedback. Some things need to make sure.
> > Currently,
> > > > > these
> > > > > > > new-added metrics follow the exact naming convention with those
> > > 'lag'
> > > > > > > counterparts, as shown below:
> > > > > > >
> > > > > > >
> > > > > > > Consumer-level metric:
> > > > > > >
> > > > > > > records-lag-max ==> records-lead-min
> > > > > > >
> > > > > > >
> > > > > > > Partition-level metrics:
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag ==>
> > > > <topic>-<partitionId>.
> > > > > > > records-lead
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-max ==>
> <topic>-<partitionId>.
> > > > > > > records-lead-min
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-avg ==>
> > <topic>-<partitionId>.
> > > > > > > records-lead-avg
> > > > > > >
> > > > > > >
> > > > > > > Correct me if I am wrong, but what you mentioned
> > `*records-lead-avg
> > > > and
> > > > > > > records-lead-min don't need the partition prefix since they are
> > > > > > aggregates
> > > > > > > across all partitions*` seemed to break the naming rule above.
> Do
> > > we
> > > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > > >
> > > > > > >
> > > > > > > huxihx
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ------------------------------
> > > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > > *收件人:* dev@kafka.apache.org
> > > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > > lead metrics to KafkaConsumer
> > > > > > >
> > > > > > > Hi, Hu,
> > > > > > >
> > > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > > mbean
> > > > > name
> > > > > > > for the new metrics? We probably want the name to be consistent
> > > with
> > > > > > > records-max-lag as described in
> > > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > seems
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > that
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > <http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Apache Kafka <http://kafka.apache.org/docum
> entation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > > records-lead-avg and records-lead-min don't need the partition
> > > prefix
> > > > > > since
> > > > > > > they are aggregates across all partitions. For records-lead, it
> > > seems
> > > > > > that
> > > > > > > it's better to add the topic partition as a tag, instead of as
> a
> > > > prefix
> > > > > > in
> > > > > > > the metric name.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > >
> > > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > > https://cwiki.apache.
> > > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> > concerning
> > > > > > adding
> > > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave
> > your
> > > > > > > comments
> > > > > > > > here. Thanks in advance.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Charly Molter
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Charly Molter
> > >
> >
>
答复: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Jun,
Sound reasonable. Already submitted a new patch<https://github.com/apache/kafka/pull/4191> and the KIP<https://cwiki.apache.org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+lead+and+per-partition+lead+metrics+to+KafkaConsumer> was also updated. Please review again. Thanks.
________________________________
发件人: Jun Rao <ju...@confluent.io>
发送时间: 2017年11月23日 6:21
收件人: dev@kafka.apache.org
主题: Re: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi, Hu,
There are two types of names. One is the sensor name, which has to be
unique. It's fine if the sensor name includes the topic/partition as the
prefix since the sensor name is only a string and is not exposed to jmx.
The second name is the metric name, which will be used in jmx. Currently,
the existing lag metric name uses topic/partition as the prefix. KIP-225
tries to change the metric name to use topic/partition as the tag. We can
just do the same thing for lead by using tags in the metric name.
Thanks,
Jun
On Mon, Nov 20, 2017 at 10:14 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi Jun,
>
>
> Seems the prefix that is used to be the unique Sensor name cannot be
> removed, so should we keep the prefix?
>
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月21日 3:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> For the new partition level metrics that you are adding, it seems it's
> better to just add the topic/partition tag instead of using them in the
> prefix. For the existing lag metrics, we can fix them in KIP-225.
>
> Thanks,
>
> Jun
>
> On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Jun,
> >
> >
> > Thanks for the comments. Do you think it'd better to add topic/partition
> > tags for those metrics as well as keep the prefix? If those prefixes
> should
> > really be removed, does this KIP need to do the same thing for `lag`
> ones?
> >
> > ________________________________
> > 发件人: Jun Rao <ju...@confluent.io>
> > 发送时间: 2017年11月18日 8:55
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition lead metrics to KafkaConsumer
> >
> > Hi, Charly,
> >
> > Thanks for the input. It makes sense.
> >
> > Hi, Hu,
> >
> > Perhaps we can keep the per partition records-lead-min and
> records-lead-avg
> > as you had before, but just add the topic and the partition as the tags
> > instead of prefix of the metric name.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
> > wrote:
> >
> > > Hi Jun, Jiangle,
> > >
> > > I'd just like to clarify that KIP-225 seems to be using per partition
> > > metric the same way as KIP-223 seems to be doing.
> > >
> > > I believe avg and max are still necessary because the MetricsReporter
> > > doesn't work in a "push" manner and the "Value" measurableStat will
> only
> > > keep the last recorded entry.
> > > Therefore a MetricsReporter usually polls to grab a current view with
> > Value
> > > this view is incomplete so it becomes not possible to compute the
> > > Max/Min/Avg.
> > > Max/Min/Avg uses SampledStats which work with a rolling window of
> samples
> > > and therefore periodic polling would work.
> > >
> > > This is why I believe it's necessary to keep Avg, Min and Max for these
> > > metrics as otherwise we wouldn't be able to recompute it in an external
> > > monitoring system.
> > >
> > > Am I wrong thinking this?
> > >
> > > Thanks,
> > > Charly
> > >
> > >
> > > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Charly,
> > > >
> > > > Thanks for KIP-225. Your proposal looks reasonable.
> > > >
> > > > Hi, Jiangjie,
> > > >
> > > > Do you think the approach that KIP-225 proposes is better for
> exposing
> > > the
> > > > per partition metric? Also, do we really need the per partition
> > > > record-lag-avg
> > > > and record-lag-max? It seems that an external monitoring system can
> > > always
> > > > derive that from the per partition record-lag.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> > charly.molter@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Jun, Hu,
> > > > >
> > > > > I have KIP-225 open for adding tags to records-lag:
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=74686649
> > > > >
> > > > > I have a patch more or less ready so I could probably get the fix
> > > checked
> > > > > in (after the vote) and you could build on top of it. Otherwise we
> > > could
> > > > > merge both KIPs if you want but they do sound different to me.
> > > > >
> > > > > Thanks!
> > > > > Charly
> > > > >
> > > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > >
> > > > > > Let me double confirm with your comments:
> > > > > >
> > > > > > 1 remove partition-level records-lead-avg and records-lead-min
> > since
> > > > they
> > > > > > both can be deduced by external monitoring system.
> > > > > >
> > > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > > >
> > > > > >
> > > > > > If they are the case you expect, do we need to do the same thing
> > for
> > > > > those
> > > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > > tagged
> > > > > > with topic&partition information which might deserve a bug.
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > > 发送时间: 2017年11月14日 12:44
> > > > > > 收件人: dev@kafka.apache.org
> > > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}".
> > > > > > So, it probably makes sense for records-lead-min to be an
> attribute
> > > > under
> > > > > > the same mbean.
> > > > > >
> > > > > > The partition level records-lead can probably be an attribute for
> > the
> > > > > mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > > where topic and partition are the tags. This matches the topic
> > level
> > > > > mbeans
> > > > > > that we have in the consumer. I am not sure what the per
> partition
> > > > level
> > > > > > records-lead-min and records-lead-avg are. Are they the min/avg
> of
> > > the
> > > > > lead
> > > > > > since the consumer is started? I am not sure we need those since
> an
> > > > > > external monitoring system can always derive them from
> > records-lead.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > > Thanks for the feedback. Some things need to make sure.
> > Currently,
> > > > > these
> > > > > > > new-added metrics follow the exact naming convention with those
> > > 'lag'
> > > > > > > counterparts, as shown below:
> > > > > > >
> > > > > > >
> > > > > > > Consumer-level metric:
> > > > > > >
> > > > > > > records-lag-max ==> records-lead-min
> > > > > > >
> > > > > > >
> > > > > > > Partition-level metrics:
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag ==>
> > > > <topic>-<partitionId>.
> > > > > > > records-lead
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-max ==>
> <topic>-<partitionId>.
> > > > > > > records-lead-min
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-avg ==>
> > <topic>-<partitionId>.
> > > > > > > records-lead-avg
> > > > > > >
> > > > > > >
> > > > > > > Correct me if I am wrong, but what you mentioned
> > `*records-lead-avg
> > > > and
> > > > > > > records-lead-min don't need the partition prefix since they are
> > > > > > aggregates
> > > > > > > across all partitions*` seemed to break the naming rule above.
> Do
> > > we
> > > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > > >
> > > > > > >
> > > > > > > huxihx
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ------------------------------
> > > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > > *收件人:* dev@kafka.apache.org
> > > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > > lead metrics to KafkaConsumer
> > > > > > >
> > > > > > > Hi, Hu,
> > > > > > >
> > > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > > mbean
> > > > > name
> > > > > > > for the new metrics? We probably want the name to be consistent
> > > with
> > > > > > > records-max-lag as described in
> > > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > seems
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> [http://apache-kafka.org/images/apache-kafka.png
[http://apache-kafka.org/images/apache-kafka.png]
]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > that
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > <http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Apache Kafka <http://kafka.apache.org/docum
> entation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > > records-lead-avg and records-lead-min don't need the partition
> > > prefix
> > > > > > since
> > > > > > > they are aggregates across all partitions. For records-lead, it
> > > seems
> > > > > > that
> > > > > > > it's better to add the topic partition as a tag, instead of as
> a
> > > > prefix
> > > > > > in
> > > > > > > the metric name.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > >
> > > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > > https://cwiki.apache.
> > > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> > concerning
> > > > > > adding
> > > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave
> > your
> > > > > > > comments
> > > > > > > > here. Thanks in advance.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Charly Molter
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Charly Molter
> > >
> >
>
Re: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and
per-partition lead metrics to KafkaConsumer
Posted by Jun Rao <ju...@confluent.io>.
Hi, Hu,
There are two types of names. One is the sensor name, which has to be
unique. It's fine if the sensor name includes the topic/partition as the
prefix since the sensor name is only a string and is not exposed to jmx.
The second name is the metric name, which will be used in jmx. Currently,
the existing lag metric name uses topic/partition as the prefix. KIP-225
tries to change the metric name to use topic/partition as the tag. We can
just do the same thing for lead by using tags in the metric name.
Thanks,
Jun
On Mon, Nov 20, 2017 at 10:14 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi Jun,
>
>
> Seems the prefix that is used to be the unique Sensor name cannot be
> removed, so should we keep the prefix?
>
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月21日 3:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> For the new partition level metrics that you are adding, it seems it's
> better to just add the topic/partition tag instead of using them in the
> prefix. For the existing lag metrics, we can fix them in KIP-225.
>
> Thanks,
>
> Jun
>
> On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Jun,
> >
> >
> > Thanks for the comments. Do you think it'd better to add topic/partition
> > tags for those metrics as well as keep the prefix? If those prefixes
> should
> > really be removed, does this KIP need to do the same thing for `lag`
> ones?
> >
> > ________________________________
> > 发件人: Jun Rao <ju...@confluent.io>
> > 发送时间: 2017年11月18日 8:55
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition lead metrics to KafkaConsumer
> >
> > Hi, Charly,
> >
> > Thanks for the input. It makes sense.
> >
> > Hi, Hu,
> >
> > Perhaps we can keep the per partition records-lead-min and
> records-lead-avg
> > as you had before, but just add the topic and the partition as the tags
> > instead of prefix of the metric name.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
> > wrote:
> >
> > > Hi Jun, Jiangle,
> > >
> > > I'd just like to clarify that KIP-225 seems to be using per partition
> > > metric the same way as KIP-223 seems to be doing.
> > >
> > > I believe avg and max are still necessary because the MetricsReporter
> > > doesn't work in a "push" manner and the "Value" measurableStat will
> only
> > > keep the last recorded entry.
> > > Therefore a MetricsReporter usually polls to grab a current view with
> > Value
> > > this view is incomplete so it becomes not possible to compute the
> > > Max/Min/Avg.
> > > Max/Min/Avg uses SampledStats which work with a rolling window of
> samples
> > > and therefore periodic polling would work.
> > >
> > > This is why I believe it's necessary to keep Avg, Min and Max for these
> > > metrics as otherwise we wouldn't be able to recompute it in an external
> > > monitoring system.
> > >
> > > Am I wrong thinking this?
> > >
> > > Thanks,
> > > Charly
> > >
> > >
> > > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Charly,
> > > >
> > > > Thanks for KIP-225. Your proposal looks reasonable.
> > > >
> > > > Hi, Jiangjie,
> > > >
> > > > Do you think the approach that KIP-225 proposes is better for
> exposing
> > > the
> > > > per partition metric? Also, do we really need the per partition
> > > > record-lag-avg
> > > > and record-lag-max? It seems that an external monitoring system can
> > > always
> > > > derive that from the per partition record-lag.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> > charly.molter@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Jun, Hu,
> > > > >
> > > > > I have KIP-225 open for adding tags to records-lag:
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=74686649
> > > > >
> > > > > I have a patch more or less ready so I could probably get the fix
> > > checked
> > > > > in (after the vote) and you could build on top of it. Otherwise we
> > > could
> > > > > merge both KIPs if you want but they do sound different to me.
> > > > >
> > > > > Thanks!
> > > > > Charly
> > > > >
> > > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > >
> > > > > > Let me double confirm with your comments:
> > > > > >
> > > > > > 1 remove partition-level records-lead-avg and records-lead-min
> > since
> > > > they
> > > > > > both can be deduced by external monitoring system.
> > > > > >
> > > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > > >
> > > > > >
> > > > > > If they are the case you expect, do we need to do the same thing
> > for
> > > > > those
> > > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > > tagged
> > > > > > with topic&partition information which might deserve a bug.
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > > 发送时间: 2017年11月14日 12:44
> > > > > > 收件人: dev@kafka.apache.org
> > > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}".
> > > > > > So, it probably makes sense for records-lead-min to be an
> attribute
> > > > under
> > > > > > the same mbean.
> > > > > >
> > > > > > The partition level records-lead can probably be an attribute for
> > the
> > > > > mbean
> > > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > > where topic and partition are the tags. This matches the topic
> > level
> > > > > mbeans
> > > > > > that we have in the consumer. I am not sure what the per
> partition
> > > > level
> > > > > > records-lead-min and records-lead-avg are. Are they the min/avg
> of
> > > the
> > > > > lead
> > > > > > since the consumer is started? I am not sure we need those since
> an
> > > > > > external monitoring system can always derive them from
> > records-lead.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > > Thanks for the feedback. Some things need to make sure.
> > Currently,
> > > > > these
> > > > > > > new-added metrics follow the exact naming convention with those
> > > 'lag'
> > > > > > > counterparts, as shown below:
> > > > > > >
> > > > > > >
> > > > > > > Consumer-level metric:
> > > > > > >
> > > > > > > records-lag-max ==> records-lead-min
> > > > > > >
> > > > > > >
> > > > > > > Partition-level metrics:
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag ==>
> > > > <topic>-<partitionId>.
> > > > > > > records-lead
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-max ==>
> <topic>-<partitionId>.
> > > > > > > records-lead-min
> > > > > > >
> > > > > > > <topic>-<partitionId>.records-lag-avg ==>
> > <topic>-<partitionId>.
> > > > > > > records-lead-avg
> > > > > > >
> > > > > > >
> > > > > > > Correct me if I am wrong, but what you mentioned
> > `*records-lead-avg
> > > > and
> > > > > > > records-lead-min don't need the partition prefix since they are
> > > > > > aggregates
> > > > > > > across all partitions*` seemed to break the naming rule above.
> Do
> > > we
> > > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > > >
> > > > > > >
> > > > > > > huxihx
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ------------------------------
> > > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > > *收件人:* dev@kafka.apache.org
> > > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > > per-partition
> > > > > > > lead metrics to KafkaConsumer
> > > > > > >
> > > > > > > Hi, Hu,
> > > > > > >
> > > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > > mbean
> > > > > name
> > > > > > > for the new metrics? We probably want the name to be consistent
> > > with
> > > > > > > records-max-lag as described in
> > > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
> [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > seems
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > that
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > <http://kafka.apache.org/documentation/#monitoring>
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Apache Kafka <http://kafka.apache.org/docum
> entation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > [http://apache-kafka.org/images/apache-kafka.png
> >
> > [http://apache-kafka.org/images/apache-kafka.png]
> >
> > ]<http:
> > > > > //kafka.apache.org/
> > > > > > documentation/#monitoring>
> > > > > >
> > > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > [http://apache-kafka.org/images/apache-kafka.png]<http://
> kafka.apache.org/
> > documentation/#monitoring>
> >
> > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> > kafka.apache.org
> > 1.2 Use Cases. Here is a description of a few of the popular use cases
> for
> > Apache Kafka®. For an overview of a number of these areas in action, see
> > this blog post.
> >
> >
> >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > kafka.apache.org
> > > > > > > 1.2 Use Cases. Here is a description of a few of the popular
> use
> > > > cases
> > > > > > for
> > > > > > > Apache Kafka®. For an overview of a number of these areas in
> > > action,
> > > > > see
> > > > > > > this blog post.
> > > > > > >
> > > > > > >
> > > > > > > records-lead-avg and records-lead-min don't need the partition
> > > prefix
> > > > > > since
> > > > > > > they are aggregates across all partitions. For records-lead, it
> > > seems
> > > > > > that
> > > > > > > it's better to add the topic partition as a tag, instead of as
> a
> > > > prefix
> > > > > > in
> > > > > > > the metric name.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > >
> > > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > > https://cwiki.apache.
> > > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> > concerning
> > > > > > adding
> > > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave
> > your
> > > > > > > comments
> > > > > > > > here. Thanks in advance.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Charly Molter
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Charly Molter
> > >
> >
>
REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
lead metrics to KafkaConsumer
Posted by Hu Xi <hu...@hotmail.com>.
Hi Jun,
Seems the prefix that is used to be the unique Sensor name cannot be removed, so should we keep the prefix?
________________________________
发件人: Jun Rao <ju...@confluent.io>
发送时间: 2017年11月21日 3:55
收件人: dev@kafka.apache.org
主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Hi, Hu,
For the new partition level metrics that you are adding, it seems it's
better to just add the topic/partition tag instead of using them in the
prefix. For the existing lag metrics, we can fix them in KIP-225.
Thanks,
Jun
On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
> Jun,
>
>
> Thanks for the comments. Do you think it'd better to add topic/partition
> tags for those metrics as well as keep the prefix? If those prefixes should
> really be removed, does this KIP need to do the same thing for `lag` ones?
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月18日 8:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Charly,
>
> Thanks for the input. It makes sense.
>
> Hi, Hu,
>
> Perhaps we can keep the per partition records-lead-min and records-lead-avg
> as you had before, but just add the topic and the partition as the tags
> instead of prefix of the metric name.
>
> Thanks,
>
> Jun
>
>
>
> On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
> wrote:
>
> > Hi Jun, Jiangle,
> >
> > I'd just like to clarify that KIP-225 seems to be using per partition
> > metric the same way as KIP-223 seems to be doing.
> >
> > I believe avg and max are still necessary because the MetricsReporter
> > doesn't work in a "push" manner and the "Value" measurableStat will only
> > keep the last recorded entry.
> > Therefore a MetricsReporter usually polls to grab a current view with
> Value
> > this view is incomplete so it becomes not possible to compute the
> > Max/Min/Avg.
> > Max/Min/Avg uses SampledStats which work with a rolling window of samples
> > and therefore periodic polling would work.
> >
> > This is why I believe it's necessary to keep Avg, Min and Max for these
> > metrics as otherwise we wouldn't be able to recompute it in an external
> > monitoring system.
> >
> > Am I wrong thinking this?
> >
> > Thanks,
> > Charly
> >
> >
> > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Charly,
> > >
> > > Thanks for KIP-225. Your proposal looks reasonable.
> > >
> > > Hi, Jiangjie,
> > >
> > > Do you think the approach that KIP-225 proposes is better for exposing
> > the
> > > per partition metric? Also, do we really need the per partition
> > > record-lag-avg
> > > and record-lag-max? It seems that an external monitoring system can
> > always
> > > derive that from the per partition record-lag.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> charly.molter@gmail.com>
> > > wrote:
> > >
> > > > Hi Jun, Hu,
> > > >
> > > > I have KIP-225 open for adding tags to records-lag:
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=74686649
> > > >
> > > > I have a patch more or less ready so I could probably get the fix
> > checked
> > > > in (after the vote) and you could build on top of it. Otherwise we
> > could
> > > > merge both KIPs if you want but they do sound different to me.
> > > >
> > > > Thanks!
> > > > Charly
> > > >
> > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > >
> > > > > Let me double confirm with your comments:
> > > > >
> > > > > 1 remove partition-level records-lead-avg and records-lead-min
> since
> > > they
> > > > > both can be deduced by external monitoring system.
> > > > >
> > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > >
> > > > >
> > > > > If they are the case you expect, do we need to do the same thing
> for
> > > > those
> > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > tagged
> > > > > with topic&partition information which might deserve a bug.
> > > > >
> > > > >
> > > > > huxihx
> > > > >
> > > > >
> > > > > ________________________________
> > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > 发送时间: 2017年11月14日 12:44
> > > > > 收件人: dev@kafka.apache.org
> > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > lead metrics to KafkaConsumer
> > > > >
> > > > > Hi, Hu,
> > > > >
> > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}".
> > > > > So, it probably makes sense for records-lead-min to be an attribute
> > > under
> > > > > the same mbean.
> > > > >
> > > > > The partition level records-lead can probably be an attribute for
> the
> > > > mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > where topic and partition are the tags. This matches the topic
> level
> > > > mbeans
> > > > > that we have in the consumer. I am not sure what the per partition
> > > level
> > > > > records-lead-min and records-lead-avg are. Are they the min/avg of
> > the
> > > > lead
> > > > > since the consumer is started? I am not sure we need those since an
> > > > > external monitoring system can always derive them from
> records-lead.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > Thanks for the feedback. Some things need to make sure.
> Currently,
> > > > these
> > > > > > new-added metrics follow the exact naming convention with those
> > 'lag'
> > > > > > counterparts, as shown below:
> > > > > >
> > > > > >
> > > > > > Consumer-level metric:
> > > > > >
> > > > > > records-lag-max ==> records-lead-min
> > > > > >
> > > > > >
> > > > > > Partition-level metrics:
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag ==>
> > > <topic>-<partitionId>.
> > > > > > records-lead
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > > > > records-lead-min
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag-avg ==>
> <topic>-<partitionId>.
> > > > > > records-lead-avg
> > > > > >
> > > > > >
> > > > > > Correct me if I am wrong, but what you mentioned
> `*records-lead-avg
> > > and
> > > > > > records-lead-min don't need the partition prefix since they are
> > > > > aggregates
> > > > > > across all partitions*` seemed to break the naming rule above. Do
> > we
> > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------
> > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > *收件人:* dev@kafka.apache.org
> > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > mbean
> > > > name
> > > > > > for the new metrics? We probably want the name to be consistent
> > with
> > > > > > records-max-lag as described in
> > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> seems
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>
Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > that
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > <http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > > records-lead-avg and records-lead-min don't need the partition
> > prefix
> > > > > since
> > > > > > they are aggregates across all partitions. For records-lead, it
> > seems
> > > > > that
> > > > > > it's better to add the topic partition as a tag, instead of as a
> > > prefix
> > > > > in
> > > > > > the metric name.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > >
> > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > https://cwiki.apache.
> > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> concerning
> > > > > adding
> > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave
> your
> > > > > > comments
> > > > > > > here. Thanks in advance.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Charly Molter
> > > >
> > >
> >
> >
> >
> > --
> > Charly Molter
> >
>
Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Posted by Jun Rao <ju...@confluent.io>.
Hi, Hu,
For the new partition level metrics that you are adding, it seems it's
better to just add the topic/partition tag instead of using them in the
prefix. For the existing lag metrics, we can fix them in KIP-225.
Thanks,
Jun
On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <hu...@hotmail.com> wrote:
> Jun,
>
>
> Thanks for the comments. Do you think it'd better to add topic/partition
> tags for those metrics as well as keep the prefix? If those prefixes should
> really be removed, does this KIP need to do the same thing for `lag` ones?
>
> ________________________________
> 发件人: Jun Rao <ju...@confluent.io>
> 发送时间: 2017年11月18日 8:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Charly,
>
> Thanks for the input. It makes sense.
>
> Hi, Hu,
>
> Perhaps we can keep the per partition records-lead-min and records-lead-avg
> as you had before, but just add the topic and the partition as the tags
> instead of prefix of the metric name.
>
> Thanks,
>
> Jun
>
>
>
> On Wed, Nov 15, 2017 at 4:58 AM, charly molter <ch...@gmail.com>
> wrote:
>
> > Hi Jun, Jiangle,
> >
> > I'd just like to clarify that KIP-225 seems to be using per partition
> > metric the same way as KIP-223 seems to be doing.
> >
> > I believe avg and max are still necessary because the MetricsReporter
> > doesn't work in a "push" manner and the "Value" measurableStat will only
> > keep the last recorded entry.
> > Therefore a MetricsReporter usually polls to grab a current view with
> Value
> > this view is incomplete so it becomes not possible to compute the
> > Max/Min/Avg.
> > Max/Min/Avg uses SampledStats which work with a rolling window of samples
> > and therefore periodic polling would work.
> >
> > This is why I believe it's necessary to keep Avg, Min and Max for these
> > metrics as otherwise we wouldn't be able to recompute it in an external
> > monitoring system.
> >
> > Am I wrong thinking this?
> >
> > Thanks,
> > Charly
> >
> >
> > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Charly,
> > >
> > > Thanks for KIP-225. Your proposal looks reasonable.
> > >
> > > Hi, Jiangjie,
> > >
> > > Do you think the approach that KIP-225 proposes is better for exposing
> > the
> > > per partition metric? Also, do we really need the per partition
> > > record-lag-avg
> > > and record-lag-max? It seems that an external monitoring system can
> > always
> > > derive that from the per partition record-lag.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> charly.molter@gmail.com>
> > > wrote:
> > >
> > > > Hi Jun, Hu,
> > > >
> > > > I have KIP-225 open for adding tags to records-lag:
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=74686649
> > > >
> > > > I have a patch more or less ready so I could probably get the fix
> > checked
> > > > in (after the vote) and you could build on top of it. Otherwise we
> > could
> > > > merge both KIPs if you want but they do sound different to me.
> > > >
> > > > Thanks!
> > > > Charly
> > > >
> > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <hu...@hotmail.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > >
> > > > > Let me double confirm with your comments:
> > > > >
> > > > > 1 remove partition-level records-lead-avg and records-lead-min
> since
> > > they
> > > > > both can be deduced by external monitoring system.
> > > > >
> > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > >
> > > > >
> > > > > If they are the case you expect, do we need to do the same thing
> for
> > > > those
> > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > tagged
> > > > > with topic&partition information which might deserve a bug.
> > > > >
> > > > >
> > > > > huxihx
> > > > >
> > > > >
> > > > > ________________________________
> > > > > 发件人: Jun Rao <ju...@confluent.io>
> > > > > 发送时间: 2017年11月14日 12:44
> > > > > 收件人: dev@kafka.apache.org
> > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > lead metrics to KafkaConsumer
> > > > >
> > > > > Hi, Hu,
> > > > >
> > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}".
> > > > > So, it probably makes sense for records-lead-min to be an attribute
> > > under
> > > > > the same mbean.
> > > > >
> > > > > The partition level records-lead can probably be an attribute for
> the
> > > > mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > where topic and partition are the tags. This matches the topic
> level
> > > > mbeans
> > > > > that we have in the consumer. I am not sure what the per partition
> > > level
> > > > > records-lead-min and records-lead-avg are. Are they the min/avg of
> > the
> > > > lead
> > > > > since the consumer is started? I am not sure we need those since an
> > > > > external monitoring system can always derive them from
> records-lead.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > Thanks for the feedback. Some things need to make sure.
> Currently,
> > > > these
> > > > > > new-added metrics follow the exact naming convention with those
> > 'lag'
> > > > > > counterparts, as shown below:
> > > > > >
> > > > > >
> > > > > > Consumer-level metric:
> > > > > >
> > > > > > records-lag-max ==> records-lead-min
> > > > > >
> > > > > >
> > > > > > Partition-level metrics:
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag ==>
> > > <topic>-<partitionId>.
> > > > > > records-lead
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > > > > records-lead-min
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag-avg ==>
> <topic>-<partitionId>.
> > > > > > records-lead-avg
> > > > > >
> > > > > >
> > > > > > Correct me if I am wrong, but what you mentioned
> `*records-lead-avg
> > > and
> > > > > > records-lead-min don't need the partition prefix since they are
> > > > > aggregates
> > > > > > across all partitions*` seemed to break the naming rule above. Do
> > we
> > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------
> > > > > > *发件人:* Jun Rao <ju...@confluent.io>
> > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > *收件人:* dev@kafka.apache.org
> > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > mbean
> > > > name
> > > > > > for the new metrics? We probably want the name to be consistent
> > with
> > > > > > records-max-lag as described in
> > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
> seems
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > that
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > <http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > > records-lead-avg and records-lead-min don't need the partition
> > prefix
> > > > > since
> > > > > > they are aggregates across all partitions. For records-lead, it
> > seems
> > > > > that
> > > > > > it's better to add the topic partition as a tag, instead of as a
> > > prefix
> > > > > in
> > > > > > the metric name.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <hu...@hotmail.com>
> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > >
> > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > https://cwiki.apache.
> > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> concerning
> > > > > adding
> > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to leave
> your
> > > > > > comments
> > > > > > > here. Thanks in advance.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Charly Molter
> > > >
> > >
> >
> >
> >
> > --
> > Charly Molter
> >
>