You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Abhishek Mendhekar <ab...@gmail.com> on 2017/06/16 16:09:21 UTC
[DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Hi Kafka Dev,
I created KIP-168 to propose adding a metric to emit total topic count
in a cluster. The metric will be emited by the controller.
The KIP can be found here
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-168%3A+Add+TotalTopicCount+metric+per+cluster)
and the assciated JIRA improvement is KAFKA-5461
(https://issues.apache.org/jira/browse/KAFKA-5461)
Appreciate all the comments.
Best,
Abhishek
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Thanks Dong!
On Wed, Jun 28, 2017 at 9:46 AM, Dong Lin <li...@gmail.com> wrote:
> Hey Abhishek,
>
> GlobalTopicCount sounds good to me.
>
> Thanks,
> Dong
>
> On Wed, Jun 28, 2017 at 8:39 AM, Abhishek Mendhekar <
> abhishek.mendhekar@gmail.com> wrote:
>
> > Hi Joel,
> >
> > Thanks for your comments.
> >
> > To keep it consistent as well as clear my suggestion is to change the
> > TopicCount to GlobalTopicCount. This makes it fairly consistent (if not
> > completely) as well as makes it unambiguous in what it means.
> >
> > Dong, Let me know if this sounds good.
> >
> > Best,
> > Abhishek
> >
> > On Mon, Jun 26, 2017 at 10:45 AM, Joel Koshy <jj...@gmail.com>
> wrote:
> >
> > > +1 on the original KIP
> > > I actually prefer TotalTopicCount because it makes it clearer that it
> is
> > a
> > > cluster-wide count. OfflinePartitionsCount is global to the cluster
> (but
> > it
> > > is fairly clear that the controller is SoT on that). TopicCount on the
> > > other hand could be misread as a local count since PartitinCount, URP,
> > are
> > > all local counts.
> > >
> > > On Thu, Jun 22, 2017 at 9:20 AM, Abhishek Mendhekar <
> > > abhishek.mendhekar@gmail.com> wrote:
> > >
> > > > Hi Kafka Dev,
> > > >
> > > > Below is the link to the update KIP proposal.
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 168%3A+Add+TopicCount+metric+per+cluster
> > > >
> > > > Thanks,
> > > > Abhishek
> > > >
> > > > On Wed, Jun 21, 2017 at 3:55 PM, Abhishek Mendhekar <
> > > > abhishek.mendhekar@gmail.com> wrote:
> > > >
> > > > > Hi Dong,
> > > > >
> > > > > Thanks for the suggestion!
> > > > >
> > > > > I think TopicCount sounds reasonable to me and it definitely seems
> > > > > consistent with the other metric names. I will update the proposal
> to
> > > > > reflect this change.
> > > > >
> > > > > Thanks,
> > > > > Abhishek
> > > > >
> > > > > On Wed, Jun 21, 2017 at 2:17 PM, Dong Lin <li...@gmail.com>
> > wrote:
> > > > >
> > > > >> Hey Abhishek,
> > > > >>
> > > > >> I think the metric is useful. Sorry for being late on this. I am
> > > > wondering
> > > > >> if TopicCount is a better name than TotalTopicCount, given that we
> > > > >> currently have metric with names OfflinePartitionsCount,
> > LeaderCount,
> > > > >> PartitionCount etc.
> > > > >>
> > > > >> Thanks,
> > > > >> Dong
> > > > >>
> > > > >> On Fri, Jun 16, 2017 at 9:09 AM, Abhishek Mendhekar <
> > > > >> abhishek.mendhekar@gmail.com> wrote:
> > > > >>
> > > > >> > Hi Kafka Dev,
> > > > >> >
> > > > >> > I created KIP-168 to propose adding a metric to emit total topic
> > > count
> > > > >> > in a cluster. The metric will be emited by the controller.
> > > > >> >
> > > > >> > The KIP can be found here
> > > > >> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > 168%3A+Add+TotalTopicCount+metric+per+cluster)
> > > > >> > and the assciated JIRA improvement is KAFKA-5461
> > > > >> > (https://issues.apache.org/jira/browse/KAFKA-5461)
> > > > >> >
> > > > >> > Appreciate all the comments.
> > > > >> >
> > > > >> > Best,
> > > > >> >
> > > > >> > Abhishek
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Abhishek Mendhekar
> > > > > abhishek.mendhekar@gmail.com | 818.263.7030 <(818)%20263-7030>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Abhishek Mendhekar
> > > > abhishek.mendhekar@gmail.com | 818.263.7030
> > > >
> > >
> >
> >
> >
> > --
> > Abhishek Mendhekar
> > abhishek.mendhekar@gmail.com | 818.263.7030
> >
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Dong Lin <li...@gmail.com>.
Hey Abhishek,
GlobalTopicCount sounds good to me.
Thanks,
Dong
On Wed, Jun 28, 2017 at 8:39 AM, Abhishek Mendhekar <
abhishek.mendhekar@gmail.com> wrote:
> Hi Joel,
>
> Thanks for your comments.
>
> To keep it consistent as well as clear my suggestion is to change the
> TopicCount to GlobalTopicCount. This makes it fairly consistent (if not
> completely) as well as makes it unambiguous in what it means.
>
> Dong, Let me know if this sounds good.
>
> Best,
> Abhishek
>
> On Mon, Jun 26, 2017 at 10:45 AM, Joel Koshy <jj...@gmail.com> wrote:
>
> > +1 on the original KIP
> > I actually prefer TotalTopicCount because it makes it clearer that it is
> a
> > cluster-wide count. OfflinePartitionsCount is global to the cluster (but
> it
> > is fairly clear that the controller is SoT on that). TopicCount on the
> > other hand could be misread as a local count since PartitinCount, URP,
> are
> > all local counts.
> >
> > On Thu, Jun 22, 2017 at 9:20 AM, Abhishek Mendhekar <
> > abhishek.mendhekar@gmail.com> wrote:
> >
> > > Hi Kafka Dev,
> > >
> > > Below is the link to the update KIP proposal.
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 168%3A+Add+TopicCount+metric+per+cluster
> > >
> > > Thanks,
> > > Abhishek
> > >
> > > On Wed, Jun 21, 2017 at 3:55 PM, Abhishek Mendhekar <
> > > abhishek.mendhekar@gmail.com> wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > Thanks for the suggestion!
> > > >
> > > > I think TopicCount sounds reasonable to me and it definitely seems
> > > > consistent with the other metric names. I will update the proposal to
> > > > reflect this change.
> > > >
> > > > Thanks,
> > > > Abhishek
> > > >
> > > > On Wed, Jun 21, 2017 at 2:17 PM, Dong Lin <li...@gmail.com>
> wrote:
> > > >
> > > >> Hey Abhishek,
> > > >>
> > > >> I think the metric is useful. Sorry for being late on this. I am
> > > wondering
> > > >> if TopicCount is a better name than TotalTopicCount, given that we
> > > >> currently have metric with names OfflinePartitionsCount,
> LeaderCount,
> > > >> PartitionCount etc.
> > > >>
> > > >> Thanks,
> > > >> Dong
> > > >>
> > > >> On Fri, Jun 16, 2017 at 9:09 AM, Abhishek Mendhekar <
> > > >> abhishek.mendhekar@gmail.com> wrote:
> > > >>
> > > >> > Hi Kafka Dev,
> > > >> >
> > > >> > I created KIP-168 to propose adding a metric to emit total topic
> > count
> > > >> > in a cluster. The metric will be emited by the controller.
> > > >> >
> > > >> > The KIP can be found here
> > > >> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> > 168%3A+Add+TotalTopicCount+metric+per+cluster)
> > > >> > and the assciated JIRA improvement is KAFKA-5461
> > > >> > (https://issues.apache.org/jira/browse/KAFKA-5461)
> > > >> >
> > > >> > Appreciate all the comments.
> > > >> >
> > > >> > Best,
> > > >> >
> > > >> > Abhishek
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Abhishek Mendhekar
> > > > abhishek.mendhekar@gmail.com | 818.263.7030 <(818)%20263-7030>
> > > >
> > >
> > >
> > >
> > > --
> > > Abhishek Mendhekar
> > > abhishek.mendhekar@gmail.com | 818.263.7030
> > >
> >
>
>
>
> --
> Abhishek Mendhekar
> abhishek.mendhekar@gmail.com | 818.263.7030
>
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Hi Joel,
Thanks for your comments.
To keep it consistent as well as clear my suggestion is to change the
TopicCount to GlobalTopicCount. This makes it fairly consistent (if not
completely) as well as makes it unambiguous in what it means.
Dong, Let me know if this sounds good.
Best,
Abhishek
On Mon, Jun 26, 2017 at 10:45 AM, Joel Koshy <jj...@gmail.com> wrote:
> +1 on the original KIP
> I actually prefer TotalTopicCount because it makes it clearer that it is a
> cluster-wide count. OfflinePartitionsCount is global to the cluster (but it
> is fairly clear that the controller is SoT on that). TopicCount on the
> other hand could be misread as a local count since PartitinCount, URP, are
> all local counts.
>
> On Thu, Jun 22, 2017 at 9:20 AM, Abhishek Mendhekar <
> abhishek.mendhekar@gmail.com> wrote:
>
> > Hi Kafka Dev,
> >
> > Below is the link to the update KIP proposal.
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 168%3A+Add+TopicCount+metric+per+cluster
> >
> > Thanks,
> > Abhishek
> >
> > On Wed, Jun 21, 2017 at 3:55 PM, Abhishek Mendhekar <
> > abhishek.mendhekar@gmail.com> wrote:
> >
> > > Hi Dong,
> > >
> > > Thanks for the suggestion!
> > >
> > > I think TopicCount sounds reasonable to me and it definitely seems
> > > consistent with the other metric names. I will update the proposal to
> > > reflect this change.
> > >
> > > Thanks,
> > > Abhishek
> > >
> > > On Wed, Jun 21, 2017 at 2:17 PM, Dong Lin <li...@gmail.com> wrote:
> > >
> > >> Hey Abhishek,
> > >>
> > >> I think the metric is useful. Sorry for being late on this. I am
> > wondering
> > >> if TopicCount is a better name than TotalTopicCount, given that we
> > >> currently have metric with names OfflinePartitionsCount, LeaderCount,
> > >> PartitionCount etc.
> > >>
> > >> Thanks,
> > >> Dong
> > >>
> > >> On Fri, Jun 16, 2017 at 9:09 AM, Abhishek Mendhekar <
> > >> abhishek.mendhekar@gmail.com> wrote:
> > >>
> > >> > Hi Kafka Dev,
> > >> >
> > >> > I created KIP-168 to propose adding a metric to emit total topic
> count
> > >> > in a cluster. The metric will be emited by the controller.
> > >> >
> > >> > The KIP can be found here
> > >> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> > 168%3A+Add+TotalTopicCount+metric+per+cluster)
> > >> > and the assciated JIRA improvement is KAFKA-5461
> > >> > (https://issues.apache.org/jira/browse/KAFKA-5461)
> > >> >
> > >> > Appreciate all the comments.
> > >> >
> > >> > Best,
> > >> >
> > >> > Abhishek
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Abhishek Mendhekar
> > > abhishek.mendhekar@gmail.com | 818.263.7030 <(818)%20263-7030>
> > >
> >
> >
> >
> > --
> > Abhishek Mendhekar
> > abhishek.mendhekar@gmail.com | 818.263.7030
> >
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Joel Koshy <jj...@gmail.com>.
+1 on the original KIP
I actually prefer TotalTopicCount because it makes it clearer that it is a
cluster-wide count. OfflinePartitionsCount is global to the cluster (but it
is fairly clear that the controller is SoT on that). TopicCount on the
other hand could be misread as a local count since PartitinCount, URP, are
all local counts.
On Thu, Jun 22, 2017 at 9:20 AM, Abhishek Mendhekar <
abhishek.mendhekar@gmail.com> wrote:
> Hi Kafka Dev,
>
> Below is the link to the update KIP proposal.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 168%3A+Add+TopicCount+metric+per+cluster
>
> Thanks,
> Abhishek
>
> On Wed, Jun 21, 2017 at 3:55 PM, Abhishek Mendhekar <
> abhishek.mendhekar@gmail.com> wrote:
>
> > Hi Dong,
> >
> > Thanks for the suggestion!
> >
> > I think TopicCount sounds reasonable to me and it definitely seems
> > consistent with the other metric names. I will update the proposal to
> > reflect this change.
> >
> > Thanks,
> > Abhishek
> >
> > On Wed, Jun 21, 2017 at 2:17 PM, Dong Lin <li...@gmail.com> wrote:
> >
> >> Hey Abhishek,
> >>
> >> I think the metric is useful. Sorry for being late on this. I am
> wondering
> >> if TopicCount is a better name than TotalTopicCount, given that we
> >> currently have metric with names OfflinePartitionsCount, LeaderCount,
> >> PartitionCount etc.
> >>
> >> Thanks,
> >> Dong
> >>
> >> On Fri, Jun 16, 2017 at 9:09 AM, Abhishek Mendhekar <
> >> abhishek.mendhekar@gmail.com> wrote:
> >>
> >> > Hi Kafka Dev,
> >> >
> >> > I created KIP-168 to propose adding a metric to emit total topic count
> >> > in a cluster. The metric will be emited by the controller.
> >> >
> >> > The KIP can be found here
> >> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > 168%3A+Add+TotalTopicCount+metric+per+cluster)
> >> > and the assciated JIRA improvement is KAFKA-5461
> >> > (https://issues.apache.org/jira/browse/KAFKA-5461)
> >> >
> >> > Appreciate all the comments.
> >> >
> >> > Best,
> >> >
> >> > Abhishek
> >> >
> >>
> >
> >
> >
> > --
> > Abhishek Mendhekar
> > abhishek.mendhekar@gmail.com | 818.263.7030 <(818)%20263-7030>
> >
>
>
>
> --
> Abhishek Mendhekar
> abhishek.mendhekar@gmail.com | 818.263.7030
>
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Hi Kafka Dev,
Below is the link to the update KIP proposal.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-168%3A+Add+TopicCount+metric+per+cluster
Thanks,
Abhishek
On Wed, Jun 21, 2017 at 3:55 PM, Abhishek Mendhekar <
abhishek.mendhekar@gmail.com> wrote:
> Hi Dong,
>
> Thanks for the suggestion!
>
> I think TopicCount sounds reasonable to me and it definitely seems
> consistent with the other metric names. I will update the proposal to
> reflect this change.
>
> Thanks,
> Abhishek
>
> On Wed, Jun 21, 2017 at 2:17 PM, Dong Lin <li...@gmail.com> wrote:
>
>> Hey Abhishek,
>>
>> I think the metric is useful. Sorry for being late on this. I am wondering
>> if TopicCount is a better name than TotalTopicCount, given that we
>> currently have metric with names OfflinePartitionsCount, LeaderCount,
>> PartitionCount etc.
>>
>> Thanks,
>> Dong
>>
>> On Fri, Jun 16, 2017 at 9:09 AM, Abhishek Mendhekar <
>> abhishek.mendhekar@gmail.com> wrote:
>>
>> > Hi Kafka Dev,
>> >
>> > I created KIP-168 to propose adding a metric to emit total topic count
>> > in a cluster. The metric will be emited by the controller.
>> >
>> > The KIP can be found here
>> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 168%3A+Add+TotalTopicCount+metric+per+cluster)
>> > and the assciated JIRA improvement is KAFKA-5461
>> > (https://issues.apache.org/jira/browse/KAFKA-5461)
>> >
>> > Appreciate all the comments.
>> >
>> > Best,
>> >
>> > Abhishek
>> >
>>
>
>
>
> --
> Abhishek Mendhekar
> abhishek.mendhekar@gmail.com | 818.263.7030 <(818)%20263-7030>
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Hi Dong,
Thanks for the suggestion!
I think TopicCount sounds reasonable to me and it definitely seems
consistent with the other metric names. I will update the proposal to
reflect this change.
Thanks,
Abhishek
On Wed, Jun 21, 2017 at 2:17 PM, Dong Lin <li...@gmail.com> wrote:
> Hey Abhishek,
>
> I think the metric is useful. Sorry for being late on this. I am wondering
> if TopicCount is a better name than TotalTopicCount, given that we
> currently have metric with names OfflinePartitionsCount, LeaderCount,
> PartitionCount etc.
>
> Thanks,
> Dong
>
> On Fri, Jun 16, 2017 at 9:09 AM, Abhishek Mendhekar <
> abhishek.mendhekar@gmail.com> wrote:
>
> > Hi Kafka Dev,
> >
> > I created KIP-168 to propose adding a metric to emit total topic count
> > in a cluster. The metric will be emited by the controller.
> >
> > The KIP can be found here
> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 168%3A+Add+TotalTopicCount+metric+per+cluster)
> > and the assciated JIRA improvement is KAFKA-5461
> > (https://issues.apache.org/jira/browse/KAFKA-5461)
> >
> > Appreciate all the comments.
> >
> > Best,
> >
> > Abhishek
> >
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Dong Lin <li...@gmail.com>.
Hey Abhishek,
I think the metric is useful. Sorry for being late on this. I am wondering
if TopicCount is a better name than TotalTopicCount, given that we
currently have metric with names OfflinePartitionsCount, LeaderCount,
PartitionCount etc.
Thanks,
Dong
On Fri, Jun 16, 2017 at 9:09 AM, Abhishek Mendhekar <
abhishek.mendhekar@gmail.com> wrote:
> Hi Kafka Dev,
>
> I created KIP-168 to propose adding a metric to emit total topic count
> in a cluster. The metric will be emited by the controller.
>
> The KIP can be found here
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 168%3A+Add+TotalTopicCount+metric+per+cluster)
> and the assciated JIRA improvement is KAFKA-5461
> (https://issues.apache.org/jira/browse/KAFKA-5461)
>
> Appreciate all the comments.
>
> Best,
>
> Abhishek
>
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Onur Karaman <on...@gmail.com>.
This metric sounds simple and useful.
+1
On Fri, Jun 16, 2017 at 9:09 AM, Abhishek Mendhekar <
abhishek.mendhekar@gmail.com> wrote:
> Hi Kafka Dev,
>
> I created KIP-168 to propose adding a metric to emit total topic count
> in a cluster. The metric will be emited by the controller.
>
> The KIP can be found here
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 168%3A+Add+TotalTopicCount+metric+per+cluster)
> and the assciated JIRA improvement is KAFKA-5461
> (https://issues.apache.org/jira/browse/KAFKA-5461)
>
> Appreciate all the comments.
>
> Best,
>
> Abhishek
>
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Hi All,
I updated the proposal to include the GlobalParitionCount metric. The
proposal for KIP-168 can be found here (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-168%3A+Add+GlobalTopicCount+and+GlobalPartitionCountmetric+per+cluster
)
Best,
Abhishek
On Mon, Jul 17, 2017 at 2:06 PM, Abhishek Mendhekar <
abhishek.mendhekar@gmail.com> wrote:
> Hi Becket,
>
> Thank you for the comments.
>
> Vahid also requested a TotalPartitionCount metric and I guess the argument
> of "aggregating this metric across all brokers in a large cluster can be
> cumbersome and inefficient" is pretty convincing. I will add this to the
> proposal and call this metric "GlobalParitionCount" (keeping it consistent
> with GlobalTopicCount).
>
> Best,
> Abhishek
>
> On Mon, Jul 17, 2017 at 1:37 PM, Becket Qin <be...@gmail.com> wrote:
>
>> Hi Abhishek,
>>
>> Sorry for the late comments. I feel that the TotalPartitionCount seems a
>> convenient metric to have. It is true that one can add the partition count
>> on each broker and get the total partition count, but it is more work and
>> in a large cluster with hundreds of brokers it seems not that efficient
>> either. Having both the total topic count and total partition count also
>> seem more symmetric.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On Wed, Jul 12, 2017 at 10:00 AM, Abhishek Mendhekar <
>> abhishek.mendhekar@gmail.com> wrote:
>>
>> > Hi Vahid,
>> >
>> > Thanks for the prompt response.
>> >
>> > Since there may be redundant metrics let's defer the total number of
>> > partition metric from the controller for now.
>> >
>> > Thanks,
>> > Abhishek
>> >
>> >
>> > On Mon, Jul 10, 2017 at 2:18 PM, Vahid S Hashemian <
>> > vahidhashemian@us.ibm.com> wrote:
>> >
>> > > Hi Abhishek,
>> > >
>> > > Thanks for your response.
>> > > Even though, as you suggested, this metric can be calculated using
>> other
>> > > present metrics, I still feel that adding the 'number of partitions'
>> > > metrics helps in better representing a Kafka cluster.
>> > > I don't feel too strong about it though (in case it is considered a
>> > > redundant metrics based on your argument).
>> > >
>> > > Thanks.
>> > > --Vahid
>> > >
>> > >
>> > >
>> > >
>> > > From: Abhishek Mendhekar <ab...@gmail.com>
>> > > To: dev@kafka.apache.org
>> > > Date: 07/10/2017 01:43 PM
>> > > Subject: Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per
>> > > cluster
>> > >
>> > >
>> > >
>> > > Hi Vahid,
>> > >
>> > > Thanks for the response.
>> > >
>> > > Kafka currently has a partition leader count metric per broker. The
>> total
>> > > partition count metric can be built by summing the partition leader
>> > > count across the cluster. This information may not be consistent but
>> > it'll
>> > > be eventually consistent in steady state.
>> > >
>> > > But if you think it'll be useful to add this metric in relation with
>> the
>> > > offline partition count to track them in tandem, then I guess it
>> might be
>> > > useful. Also, this will be consistent as it is coming from the
>> > controller.
>> > > Let me know your thoughts on this.
>> > >
>> > > Best,
>> > > Abhishek
>> > >
>> > >
>> > > On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
>> > > vahidhashemian@us.ibm.com> wrote:
>> > >
>> > > > Hi Abhishek,
>> > > >
>> > > > Thanks for the KIP and sorry for posting late on this thread.
>> > > >
>> > > > The proposal looks good to me.
>> > > > I wonder if it would be useful to also add a partition count metric.
>> > > > It seems there is already an offline partition count metric in the
>> > > > controller, but no total partition count.
>> > > >
>> > > > Thanks.
>> > > > --Vahid
>> > > >
>> > > >
>> > > >
>> > > > From: Abhishek Mendhekar <ab...@gmail.com>
>> > > > To: dev@kafka.apache.org
>> > > > Date: 06/16/2017 09:09 AM
>> > > > Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per
>> > > cluster
>> > > >
>> > > >
>> > > >
>> > > > Hi Kafka Dev,
>> > > >
>> > > > I created KIP-168 to propose adding a metric to emit total topic
>> count
>> > > > in a cluster. The metric will be emited by the controller.
>> > > >
>> > > > The KIP can be found here
>> > > > (
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > 168%3A+Add+TotalTopicCount+metric+per+cluster
>> > > > )
>> > > > and the assciated JIRA improvement is KAFKA-5461
>> > > > (https://issues.apache.org/jira/browse/KAFKA-5461)
>> > > >
>> > > > Appreciate all the comments.
>> > > >
>> > > > Best,
>> > > >
>> > > > Abhishek
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Abhishek Mendhekar
>> > > abhishek.mendhekar@gmail.com | 818.263.7030
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> > --
>> > Abhishek Mendhekar
>> > abhishek.mendhekar@gmail.com | 818.263.7030
>> >
>>
>
>
>
> --
> Abhishek Mendhekar
> abhishek.mendhekar@gmail.com | 818.263.7030 <(818)%20263-7030>
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Hi Becket,
Thank you for the comments.
Vahid also requested a TotalPartitionCount metric and I guess the argument
of "aggregating this metric across all brokers in a large cluster can be
cumbersome and inefficient" is pretty convincing. I will add this to the
proposal and call this metric "GlobalParitionCount" (keeping it consistent
with GlobalTopicCount).
Best,
Abhishek
On Mon, Jul 17, 2017 at 1:37 PM, Becket Qin <be...@gmail.com> wrote:
> Hi Abhishek,
>
> Sorry for the late comments. I feel that the TotalPartitionCount seems a
> convenient metric to have. It is true that one can add the partition count
> on each broker and get the total partition count, but it is more work and
> in a large cluster with hundreds of brokers it seems not that efficient
> either. Having both the total topic count and total partition count also
> seem more symmetric.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Jul 12, 2017 at 10:00 AM, Abhishek Mendhekar <
> abhishek.mendhekar@gmail.com> wrote:
>
> > Hi Vahid,
> >
> > Thanks for the prompt response.
> >
> > Since there may be redundant metrics let's defer the total number of
> > partition metric from the controller for now.
> >
> > Thanks,
> > Abhishek
> >
> >
> > On Mon, Jul 10, 2017 at 2:18 PM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Abhishek,
> > >
> > > Thanks for your response.
> > > Even though, as you suggested, this metric can be calculated using
> other
> > > present metrics, I still feel that adding the 'number of partitions'
> > > metrics helps in better representing a Kafka cluster.
> > > I don't feel too strong about it though (in case it is considered a
> > > redundant metrics based on your argument).
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Abhishek Mendhekar <ab...@gmail.com>
> > > To: dev@kafka.apache.org
> > > Date: 07/10/2017 01:43 PM
> > > Subject: Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> > > cluster
> > >
> > >
> > >
> > > Hi Vahid,
> > >
> > > Thanks for the response.
> > >
> > > Kafka currently has a partition leader count metric per broker. The
> total
> > > partition count metric can be built by summing the partition leader
> > > count across the cluster. This information may not be consistent but
> > it'll
> > > be eventually consistent in steady state.
> > >
> > > But if you think it'll be useful to add this metric in relation with
> the
> > > offline partition count to track them in tandem, then I guess it might
> be
> > > useful. Also, this will be consistent as it is coming from the
> > controller.
> > > Let me know your thoughts on this.
> > >
> > > Best,
> > > Abhishek
> > >
> > >
> > > On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Abhishek,
> > > >
> > > > Thanks for the KIP and sorry for posting late on this thread.
> > > >
> > > > The proposal looks good to me.
> > > > I wonder if it would be useful to also add a partition count metric.
> > > > It seems there is already an offline partition count metric in the
> > > > controller, but no total partition count.
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > > From: Abhishek Mendhekar <ab...@gmail.com>
> > > > To: dev@kafka.apache.org
> > > > Date: 06/16/2017 09:09 AM
> > > > Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> > > cluster
> > > >
> > > >
> > > >
> > > > Hi Kafka Dev,
> > > >
> > > > I created KIP-168 to propose adding a metric to emit total topic
> count
> > > > in a cluster. The metric will be emited by the controller.
> > > >
> > > > The KIP can be found here
> > > > (
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 168%3A+Add+TotalTopicCount+metric+per+cluster
> > > > )
> > > > and the assciated JIRA improvement is KAFKA-5461
> > > > (https://issues.apache.org/jira/browse/KAFKA-5461)
> > > >
> > > > Appreciate all the comments.
> > > >
> > > > Best,
> > > >
> > > > Abhishek
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Abhishek Mendhekar
> > > abhishek.mendhekar@gmail.com | 818.263.7030
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Abhishek Mendhekar
> > abhishek.mendhekar@gmail.com | 818.263.7030
> >
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Becket Qin <be...@gmail.com>.
Hi Abhishek,
Sorry for the late comments. I feel that the TotalPartitionCount seems a
convenient metric to have. It is true that one can add the partition count
on each broker and get the total partition count, but it is more work and
in a large cluster with hundreds of brokers it seems not that efficient
either. Having both the total topic count and total partition count also
seem more symmetric.
Thanks,
Jiangjie (Becket) Qin
On Wed, Jul 12, 2017 at 10:00 AM, Abhishek Mendhekar <
abhishek.mendhekar@gmail.com> wrote:
> Hi Vahid,
>
> Thanks for the prompt response.
>
> Since there may be redundant metrics let's defer the total number of
> partition metric from the controller for now.
>
> Thanks,
> Abhishek
>
>
> On Mon, Jul 10, 2017 at 2:18 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Abhishek,
> >
> > Thanks for your response.
> > Even though, as you suggested, this metric can be calculated using other
> > present metrics, I still feel that adding the 'number of partitions'
> > metrics helps in better representing a Kafka cluster.
> > I don't feel too strong about it though (in case it is considered a
> > redundant metrics based on your argument).
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From: Abhishek Mendhekar <ab...@gmail.com>
> > To: dev@kafka.apache.org
> > Date: 07/10/2017 01:43 PM
> > Subject: Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> > cluster
> >
> >
> >
> > Hi Vahid,
> >
> > Thanks for the response.
> >
> > Kafka currently has a partition leader count metric per broker. The total
> > partition count metric can be built by summing the partition leader
> > count across the cluster. This information may not be consistent but
> it'll
> > be eventually consistent in steady state.
> >
> > But if you think it'll be useful to add this metric in relation with the
> > offline partition count to track them in tandem, then I guess it might be
> > useful. Also, this will be consistent as it is coming from the
> controller.
> > Let me know your thoughts on this.
> >
> > Best,
> > Abhishek
> >
> >
> > On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Abhishek,
> > >
> > > Thanks for the KIP and sorry for posting late on this thread.
> > >
> > > The proposal looks good to me.
> > > I wonder if it would be useful to also add a partition count metric.
> > > It seems there is already an offline partition count metric in the
> > > controller, but no total partition count.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > > From: Abhishek Mendhekar <ab...@gmail.com>
> > > To: dev@kafka.apache.org
> > > Date: 06/16/2017 09:09 AM
> > > Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> > cluster
> > >
> > >
> > >
> > > Hi Kafka Dev,
> > >
> > > I created KIP-168 to propose adding a metric to emit total topic count
> > > in a cluster. The metric will be emited by the controller.
> > >
> > > The KIP can be found here
> > > (
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 168%3A+Add+TotalTopicCount+metric+per+cluster
> > > )
> > > and the assciated JIRA improvement is KAFKA-5461
> > > (https://issues.apache.org/jira/browse/KAFKA-5461)
> > >
> > > Appreciate all the comments.
> > >
> > > Best,
> > >
> > > Abhishek
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Abhishek Mendhekar
> > abhishek.mendhekar@gmail.com | 818.263.7030
> >
> >
> >
> >
> >
>
>
> --
> Abhishek Mendhekar
> abhishek.mendhekar@gmail.com | 818.263.7030
>
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Hi Vahid,
Thanks for the prompt response.
Since there may be redundant metrics let's defer the total number of
partition metric from the controller for now.
Thanks,
Abhishek
On Mon, Jul 10, 2017 at 2:18 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Abhishek,
>
> Thanks for your response.
> Even though, as you suggested, this metric can be calculated using other
> present metrics, I still feel that adding the 'number of partitions'
> metrics helps in better representing a Kafka cluster.
> I don't feel too strong about it though (in case it is considered a
> redundant metrics based on your argument).
>
> Thanks.
> --Vahid
>
>
>
>
> From: Abhishek Mendhekar <ab...@gmail.com>
> To: dev@kafka.apache.org
> Date: 07/10/2017 01:43 PM
> Subject: Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> cluster
>
>
>
> Hi Vahid,
>
> Thanks for the response.
>
> Kafka currently has a partition leader count metric per broker. The total
> partition count metric can be built by summing the partition leader
> count across the cluster. This information may not be consistent but it'll
> be eventually consistent in steady state.
>
> But if you think it'll be useful to add this metric in relation with the
> offline partition count to track them in tandem, then I guess it might be
> useful. Also, this will be consistent as it is coming from the controller.
> Let me know your thoughts on this.
>
> Best,
> Abhishek
>
>
> On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Abhishek,
> >
> > Thanks for the KIP and sorry for posting late on this thread.
> >
> > The proposal looks good to me.
> > I wonder if it would be useful to also add a partition count metric.
> > It seems there is already an offline partition count metric in the
> > controller, but no total partition count.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From: Abhishek Mendhekar <ab...@gmail.com>
> > To: dev@kafka.apache.org
> > Date: 06/16/2017 09:09 AM
> > Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per
> cluster
> >
> >
> >
> > Hi Kafka Dev,
> >
> > I created KIP-168 to propose adding a metric to emit total topic count
> > in a cluster. The metric will be emited by the controller.
> >
> > The KIP can be found here
> > (
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 168%3A+Add+TotalTopicCount+metric+per+cluster
> > )
> > and the assciated JIRA improvement is KAFKA-5461
> > (https://issues.apache.org/jira/browse/KAFKA-5461)
> >
> > Appreciate all the comments.
> >
> > Best,
> >
> > Abhishek
> >
> >
> >
> >
> >
>
>
> --
> Abhishek Mendhekar
> abhishek.mendhekar@gmail.com | 818.263.7030
>
>
>
>
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Abhishek,
Thanks for your response.
Even though, as you suggested, this metric can be calculated using other
present metrics, I still feel that adding the 'number of partitions'
metrics helps in better representing a Kafka cluster.
I don't feel too strong about it though (in case it is considered a
redundant metrics based on your argument).
Thanks.
--Vahid
From: Abhishek Mendhekar <ab...@gmail.com>
To: dev@kafka.apache.org
Date: 07/10/2017 01:43 PM
Subject: Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per
cluster
Hi Vahid,
Thanks for the response.
Kafka currently has a partition leader count metric per broker. The total
partition count metric can be built by summing the partition leader
count across the cluster. This information may not be consistent but it'll
be eventually consistent in steady state.
But if you think it'll be useful to add this metric in relation with the
offline partition count to track them in tandem, then I guess it might be
useful. Also, this will be consistent as it is coming from the controller.
Let me know your thoughts on this.
Best,
Abhishek
On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Abhishek,
>
> Thanks for the KIP and sorry for posting late on this thread.
>
> The proposal looks good to me.
> I wonder if it would be useful to also add a partition count metric.
> It seems there is already an offline partition count metric in the
> controller, but no total partition count.
>
> Thanks.
> --Vahid
>
>
>
> From: Abhishek Mendhekar <ab...@gmail.com>
> To: dev@kafka.apache.org
> Date: 06/16/2017 09:09 AM
> Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per
cluster
>
>
>
> Hi Kafka Dev,
>
> I created KIP-168 to propose adding a metric to emit total topic count
> in a cluster. The metric will be emited by the controller.
>
> The KIP can be found here
> (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 168%3A+Add+TotalTopicCount+metric+per+cluster
> )
> and the assciated JIRA improvement is KAFKA-5461
> (https://issues.apache.org/jira/browse/KAFKA-5461)
>
> Appreciate all the comments.
>
> Best,
>
> Abhishek
>
>
>
>
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Abhishek Mendhekar <ab...@gmail.com>.
Hi Vahid,
Thanks for the response.
Kafka currently has a partition leader count metric per broker. The total
partition count metric can be built by summing the partition leader
count across the cluster. This information may not be consistent but it'll
be eventually consistent in steady state.
But if you think it'll be useful to add this metric in relation with the
offline partition count to track them in tandem, then I guess it might be
useful. Also, this will be consistent as it is coming from the controller.
Let me know your thoughts on this.
Best,
Abhishek
On Wed, Jun 28, 2017 at 9:05 AM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Abhishek,
>
> Thanks for the KIP and sorry for posting late on this thread.
>
> The proposal looks good to me.
> I wonder if it would be useful to also add a partition count metric.
> It seems there is already an offline partition count metric in the
> controller, but no total partition count.
>
> Thanks.
> --Vahid
>
>
>
> From: Abhishek Mendhekar <ab...@gmail.com>
> To: dev@kafka.apache.org
> Date: 06/16/2017 09:09 AM
> Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
>
>
>
> Hi Kafka Dev,
>
> I created KIP-168 to propose adding a metric to emit total topic count
> in a cluster. The metric will be emited by the controller.
>
> The KIP can be found here
> (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 168%3A+Add+TotalTopicCount+metric+per+cluster
> )
> and the assciated JIRA improvement is KAFKA-5461
> (https://issues.apache.org/jira/browse/KAFKA-5461)
>
> Appreciate all the comments.
>
> Best,
>
> Abhishek
>
>
>
>
>
--
Abhishek Mendhekar
abhishek.mendhekar@gmail.com | 818.263.7030
Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Abhishek,
Thanks for the KIP and sorry for posting late on this thread.
The proposal looks good to me.
I wonder if it would be useful to also add a partition count metric.
It seems there is already an offline partition count metric in the
controller, but no total partition count.
Thanks.
--Vahid
From: Abhishek Mendhekar <ab...@gmail.com>
To: dev@kafka.apache.org
Date: 06/16/2017 09:09 AM
Subject: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster
Hi Kafka Dev,
I created KIP-168 to propose adding a metric to emit total topic count
in a cluster. The metric will be emited by the controller.
The KIP can be found here
(
https://cwiki.apache.org/confluence/display/KAFKA/KIP-168%3A+Add+TotalTopicCount+metric+per+cluster
)
and the assciated JIRA improvement is KAFKA-5461
(https://issues.apache.org/jira/browse/KAFKA-5461)
Appreciate all the comments.
Best,
Abhishek