You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Abhishek Mendhekar <ab...@gmail.com> on 2017/07/10 20:35:28 UTC
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Hi Vahid,
Thanks for the response.
Kafka currently has a partition leader count metric per broker. The total
partition count metric can be built by summing the partition leader
count across the cluster. This information may not be consistent but it'll
be eventually consistent in steady state.
But if you think it'll be useful to add this metric in relation with the
offline partition count to track them in tandem, then I guess it might be
useful. Also, this will be consistent as it is coming from the controller.
Let me know your thoughts on this.
Best,
Abhishek
On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Abhishek,
>
> Thanks for the KIP and sorry for posting late on this thread.
>
> The proposal looks good to me.
> I wonder if it would be useful to also add a partition count metric.
> It seems there is already an offline partition count metric in the
> controller, but no total partition count.
>
> Thanks.
> --Vahid
>
>
>
> From: Abhishek Mendhekar <ab...@gmail.com>
> To: dev@kafka.apache.org
> Date: 06/16/2017 09:09 AM
> Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
>
>
>
> Hi Kafka Dev,
>
> I created KIP-168 to propose adding a metric to emit total topic count
> in a cluster. The metric will be emited by the controller.
>
> The KIP can be found here
> (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 168%3A+Add+TotalTopicCount+metric+per+cluster
> )
> and the assciated JIRA improvement is KAFKA-5461
> (https://issues.apache.org/jira/browse/KAFKA-5461)
>
> Appreciate all the comments.
>
> Best,
>
> Abhishek
>
>
>
>
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Hi All,
I updated the proposal to include the GlobalParitionCount metric. The
proposal for KIP-168 can be found here (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-168%3A+Add+GlobalTopicCount+and+GlobalPartitionCountmetric+per+cluster
)
Best,
Abhishek
On Mon, Jul 17, 2017 at 2:06 PM, Abhishek Mendhekar <
abhishek.mendhekar@gmail.com> wrote:
> Hi Becket,
>
> Thank you for the comments.
>
> Vahid also requested a TotalPartitionCount metric and I guess the argument
> of "aggregating this metric across all brokers in a large cluster can be
> cumbersome and inefficient" is pretty convincing. I will add this to the
> proposal and call this metric "GlobalParitionCount" (keeping it consistent
> with GlobalTopicCount).
>
> Best,
> Abhishek
>
> On Mon, Jul 17, 2017 at 1:37 PM, Becket Qin <be...@gmail.com> wrote:
>
>> Hi Abhishek,
>>
>> Sorry for the late comments. I feel that the TotalPartitionCount seems a
>> convenient metric to have. It is true that one can add the partition count
>> on each broker and get the total partition count, but it is more work and
>> in a large cluster with hundreds of brokers it seems not that efficient
>> either. Having both the total topic count and total partition count also
>> seem more symmetric.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On Wed, Jul 12, 2017 at 10:00 AM, Abhishek Mendhekar <
>> abhishek.mendhekar@gmail.com> wrote:
>>
>> > Hi Vahid,
>> >
>> > Thanks for the prompt response.
>> >
>> > Since there may be redundant metrics let's defer the total number of
>> > partition metric from the controller for now.
>> >
>> > Thanks,
>> > Abhishek
>> >
>> >
>> > On Mon, Jul 10, 2017 at 2:18 PM, Vahid S Hashemian <
>> > vahidhashemian@us.ibm.com> wrote:
>> >
>> > > Hi Abhishek,
>> > >
>> > > Thanks for your response.
>> > > Even though, as you suggested, this metric can be calculated using
>> other
>> > > present metrics, I still feel that adding the 'number of partitions'
>> > > metrics helps in better representing a Kafka cluster.
>> > > I don't feel too strong about it though (in case it is considered a
>> > > redundant metrics based on your argument).
>> > >
>> > > Thanks.
>> > > --Vahid
>> > >
>> > >
>> > >
>> > >
>> > > From: Abhishek Mendhekar <ab...@gmail.com>
>> > > To: dev@kafka.apache.org
>> > > Date: 07/10/2017 01:43 PM
>> > > Subject: Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per
>> > > cluster
>> > >
>> > >
>> > >
>> > > Hi Vahid,
>> > >
>> > > Thanks for the response.
>> > >
>> > > Kafka currently has a partition leader count metric per broker. The
>> total
>> > > partition count metric can be built by summing the partition leader
>> > > count across the cluster. This information may not be consistent but
>> > it'll
>> > > be eventually consistent in steady state.
>> > >
>> > > But if you think it'll be useful to add this metric in relation with
>> the
>> > > offline partition count to track them in tandem, then I guess it
>> might be
>> > > useful. Also, this will be consistent as it is coming from the
>> > controller.
>> > > Let me know your thoughts on this.
>> > >
>> > > Best,
>> > > Abhishek
>> > >
>> > >
>> > > On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
>> > > vahidhashemian@us.ibm.com> wrote:
>> > >
>> > > > Hi Abhishek,
>> > > >
>> > > > Thanks for the KIP and sorry for posting late on this thread.
>> > > >
>> > > > The proposal looks good to me.
>> > > > I wonder if it would be useful to also add a partition count metric.
>> > > > It seems there is already an offline partition count metric in the
>> > > > controller, but no total partition count.
>> > > >
>> > > > Thanks.
>> > > > --Vahid
>> > > >
>> > > >
>> > > >
>> > > > From: Abhishek Mendhekar <ab...@gmail.com>
>> > > > To: dev@kafka.apache.org
>> > > > Date: 06/16/2017 09:09 AM
>> > > > Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per
>> > > cluster
>> > > >
>> > > >
>> > > >
>> > > > Hi Kafka Dev,
>> > > >
>> > > > I created KIP-168 to propose adding a metric to emit total topic
>> count
>> > > > in a cluster. The metric will be emited by the controller.
>> > > >
>> > > > The KIP can be found here
>> > > > (
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > 168%3A+Add+TotalTopicCount+metric+per+cluster
>> > > > )
>> > > > and the assciated JIRA improvement is KAFKA-5461
>> > > > (https://issues.apache.org/jira/browse/KAFKA-5461)
>> > > >
>> > > > Appreciate all the comments.
>> > > >
>> > > > Best,
>> > > >
>> > > > Abhishek
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Abhishek Mendhekar
>> > > abhishek.mendhekar@gmail.com | 818.263.7030
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> > --
>> > Abhishek Mendhekar
>> > abhishek.mendhekar@gmail.com | 818.263.7030
>> >
>>
>
>
>
> --
> Abhishek Mendhekar
> abhishek.mendhekar@gmail.com | 818.263.7030 <(818)%20263-7030>
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Hi Becket,
Thank you for the comments.
Vahid also requested a TotalPartitionCount metric and I guess the argument
of "aggregating this metric across all brokers in a large cluster can be
cumbersome and inefficient" is pretty convincing. I will add this to the
proposal and call this metric "GlobalParitionCount" (keeping it consistent
with GlobalTopicCount).
Best,
Abhishek
On Mon, Jul 17, 2017 at 1:37 PM, Becket Qin <be...@gmail.com> wrote:
> Hi Abhishek,
>
> Sorry for the late comments. I feel that the TotalPartitionCount seems a
> convenient metric to have. It is true that one can add the partition count
> on each broker and get the total partition count, but it is more work and
> in a large cluster with hundreds of brokers it seems not that efficient
> either. Having both the total topic count and total partition count also
> seem more symmetric.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Jul 12, 2017 at 10:00 AM, Abhishek Mendhekar <
> abhishek.mendhekar@gmail.com> wrote:
>
> > Hi Vahid,
> >
> > Thanks for the prompt response.
> >
> > Since there may be redundant metrics let's defer the total number of
> > partition metric from the controller for now.
> >
> > Thanks,
> > Abhishek
> >
> >
> > On Mon, Jul 10, 2017 at 2:18 PM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Abhishek,
> > >
> > > Thanks for your response.
> > > Even though, as you suggested, this metric can be calculated using
> other
> > > present metrics, I still feel that adding the 'number of partitions'
> > > metrics helps in better representing a Kafka cluster.
> > > I don't feel too strong about it though (in case it is considered a
> > > redundant metrics based on your argument).
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Abhishek Mendhekar <ab...@gmail.com>
> > > To: dev@kafka.apache.org
> > > Date: 07/10/2017 01:43 PM
> > > Subject: Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> > > cluster
> > >
> > >
> > >
> > > Hi Vahid,
> > >
> > > Thanks for the response.
> > >
> > > Kafka currently has a partition leader count metric per broker. The
> total
> > > partition count metric can be built by summing the partition leader
> > > count across the cluster. This information may not be consistent but
> > it'll
> > > be eventually consistent in steady state.
> > >
> > > But if you think it'll be useful to add this metric in relation with
> the
> > > offline partition count to track them in tandem, then I guess it might
> be
> > > useful. Also, this will be consistent as it is coming from the
> > controller.
> > > Let me know your thoughts on this.
> > >
> > > Best,
> > > Abhishek
> > >
> > >
> > > On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Abhishek,
> > > >
> > > > Thanks for the KIP and sorry for posting late on this thread.
> > > >
> > > > The proposal looks good to me.
> > > > I wonder if it would be useful to also add a partition count metric.
> > > > It seems there is already an offline partition count metric in the
> > > > controller, but no total partition count.
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > > From: Abhishek Mendhekar <ab...@gmail.com>
> > > > To: dev@kafka.apache.org
> > > > Date: 06/16/2017 09:09 AM
> > > > Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> > > cluster
> > > >
> > > >
> > > >
> > > > Hi Kafka Dev,
> > > >
> > > > I created KIP-168 to propose adding a metric to emit total topic
> count
> > > > in a cluster. The metric will be emited by the controller.
> > > >
> > > > The KIP can be found here
> > > > (
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 168%3A+Add+TotalTopicCount+metric+per+cluster
> > > > )
> > > > and the assciated JIRA improvement is KAFKA-5461
> > > > (https://issues.apache.org/jira/browse/KAFKA-5461)
> > > >
> > > > Appreciate all the comments.
> > > >
> > > > Best,
> > > >
> > > > Abhishek
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Abhishek Mendhekar
> > > abhishek.mendhekar@gmail.com | 818.263.7030
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Abhishek Mendhekar
> > abhishek.mendhekar@gmail.com | 818.263.7030
> >
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Becket Qin <be...@gmail.com>.
Hi Abhishek,
Sorry for the late comments. I feel that the TotalPartitionCount seems a
convenient metric to have. It is true that one can add the partition count
on each broker and get the total partition count, but it is more work and
in a large cluster with hundreds of brokers it seems not that efficient
either. Having both the total topic count and total partition count also
seem more symmetric.
Thanks,
Jiangjie (Becket) Qin
On Wed, Jul 12, 2017 at 10:00 AM, Abhishek Mendhekar <
abhishek.mendhekar@gmail.com> wrote:
> Hi Vahid,
>
> Thanks for the prompt response.
>
> Since there may be redundant metrics let's defer the total number of
> partition metric from the controller for now.
>
> Thanks,
> Abhishek
>
>
> On Mon, Jul 10, 2017 at 2:18 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Abhishek,
> >
> > Thanks for your response.
> > Even though, as you suggested, this metric can be calculated using other
> > present metrics, I still feel that adding the 'number of partitions'
> > metrics helps in better representing a Kafka cluster.
> > I don't feel too strong about it though (in case it is considered a
> > redundant metrics based on your argument).
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From: Abhishek Mendhekar <ab...@gmail.com>
> > To: dev@kafka.apache.org
> > Date: 07/10/2017 01:43 PM
> > Subject: Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> > cluster
> >
> >
> >
> > Hi Vahid,
> >
> > Thanks for the response.
> >
> > Kafka currently has a partition leader count metric per broker. The total
> > partition count metric can be built by summing the partition leader
> > count across the cluster. This information may not be consistent but
> it'll
> > be eventually consistent in steady state.
> >
> > But if you think it'll be useful to add this metric in relation with the
> > offline partition count to track them in tandem, then I guess it might be
> > useful. Also, this will be consistent as it is coming from the
> controller.
> > Let me know your thoughts on this.
> >
> > Best,
> > Abhishek
> >
> >
> > On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Abhishek,
> > >
> > > Thanks for the KIP and sorry for posting late on this thread.
> > >
> > > The proposal looks good to me.
> > > I wonder if it would be useful to also add a partition count metric.
> > > It seems there is already an offline partition count metric in the
> > > controller, but no total partition count.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > > From: Abhishek Mendhekar <ab...@gmail.com>
> > > To: dev@kafka.apache.org
> > > Date: 06/16/2017 09:09 AM
> > > Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> > cluster
> > >
> > >
> > >
> > > Hi Kafka Dev,
> > >
> > > I created KIP-168 to propose adding a metric to emit total topic count
> > > in a cluster. The metric will be emited by the controller.
> > >
> > > The KIP can be found here
> > > (
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 168%3A+Add+TotalTopicCount+metric+per+cluster
> > > )
> > > and the assciated JIRA improvement is KAFKA-5461
> > > (https://issues.apache.org/jira/browse/KAFKA-5461)
> > >
> > > Appreciate all the comments.
> > >
> > > Best,
> > >
> > > Abhishek
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Abhishek Mendhekar
> > abhishek.mendhekar@gmail.com | 818.263.7030
> >
> >
> >
> >
> >
>
>
> --
> Abhishek Mendhekar
> abhishek.mendhekar@gmail.com | 818.263.7030
>
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Hi Vahid,
Thanks for the prompt response.
Since there may be redundant metrics let's defer the total number of
partition metric from the controller for now.
Thanks,
Abhishek
On Mon, Jul 10, 2017 at 2:18 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Abhishek,
>
> Thanks for your response.
> Even though, as you suggested, this metric can be calculated using other
> present metrics, I still feel that adding the 'number of partitions'
> metrics helps in better representing a Kafka cluster.
> I don't feel too strong about it though (in case it is considered a
> redundant metrics based on your argument).
>
> Thanks.
> --Vahid
>
>
>
>
> From: Abhishek Mendhekar <ab...@gmail.com>
> To: dev@kafka.apache.org
> Date: 07/10/2017 01:43 PM
> Subject: Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> cluster
>
>
>
> Hi Vahid,
>
> Thanks for the response.
>
> Kafka currently has a partition leader count metric per broker. The total
> partition count metric can be built by summing the partition leader
> count across the cluster. This information may not be consistent but it'll
> be eventually consistent in steady state.
>
> But if you think it'll be useful to add this metric in relation with the
> offline partition count to track them in tandem, then I guess it might be
> useful. Also, this will be consistent as it is coming from the controller.
> Let me know your thoughts on this.
>
> Best,
> Abhishek
>
>
> On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Abhishek,
> >
> > Thanks for the KIP and sorry for posting late on this thread.
> >
> > The proposal looks good to me.
> > I wonder if it would be useful to also add a partition count metric.
> > It seems there is already an offline partition count metric in the
> > controller, but no total partition count.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From: Abhishek Mendhekar <ab...@gmail.com>
> > To: dev@kafka.apache.org
> > Date: 06/16/2017 09:09 AM
> > Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> cluster
> >
> >
> >
> > Hi Kafka Dev,
> >
> > I created KIP-168 to propose adding a metric to emit total topic count
> > in a cluster. The metric will be emited by the controller.
> >
> > The KIP can be found here
> > (
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 168%3A+Add+TotalTopicCount+metric+per+cluster
> > )
> > and the assciated JIRA improvement is KAFKA-5461
> > (https://issues.apache.org/jira/browse/KAFKA-5461)
> >
> > Appreciate all the comments.
> >
> > Best,
> >
> > Abhishek
> >
> >
> >
> >
> >
>
>
> --
> Abhishek Mendhekar
> abhishek.mendhekar@gmail.com | 818.263.7030
>
>
>
>
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Abhishek,
Thanks for your response.
Even though, as you suggested, this metric can be calculated using other
present metrics, I still feel that adding the 'number of partitions'
metrics helps in better representing a Kafka cluster.
I don't feel too strong about it though (in case it is considered a
redundant metrics based on your argument).
Thanks.
--Vahid
From: Abhishek Mendhekar <ab...@gmail.com>
To: dev@kafka.apache.org
Date: 07/10/2017 01:43 PM
Subject: Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per
cluster
Hi Vahid,
Thanks for the response.
Kafka currently has a partition leader count metric per broker. The total
partition count metric can be built by summing the partition leader
count across the cluster. This information may not be consistent but it'll
be eventually consistent in steady state.
But if you think it'll be useful to add this metric in relation with the
offline partition count to track them in tandem, then I guess it might be
useful. Also, this will be consistent as it is coming from the controller.
Let me know your thoughts on this.
Best,
Abhishek
On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Abhishek,
>
> Thanks for the KIP and sorry for posting late on this thread.
>
> The proposal looks good to me.
> I wonder if it would be useful to also add a partition count metric.
> It seems there is already an offline partition count metric in the
> controller, but no total partition count.
>
> Thanks.
> --Vahid
>
>
>
> From: Abhishek Mendhekar <ab...@gmail.com>
> To: dev@kafka.apache.org
> Date: 06/16/2017 09:09 AM
> Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per
cluster
>
>
>
> Hi Kafka Dev,
>
> I created KIP-168 to propose adding a metric to emit total topic count
> in a cluster. The metric will be emited by the controller.
>
> The KIP can be found here
> (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 168%3A+Add+TotalTopicCount+metric+per+cluster
> )
> and the assciated JIRA improvement is KAFKA-5461
> (https://issues.apache.org/jira/browse/KAFKA-5461)
>
> Appreciate all the comments.
>
> Best,
>
> Abhishek
>
>
>
>
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030