You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Chia-Ping Tsai <ch...@apache.org> on 2020/03/18 17:48:21 UTC
[DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker
hi
this ticket is about to records partition metrics rather than topic metrics. It helps us to observe more precis metrics for specify partition. The downside is that broker needs more space to keep metrics and the origin metrics interface (tags) is broken since this ticket adds new tag "partition=xxx" to it. The alternative is an new config flag used to enable new metrics (of course, it is false by default). Also, we ought to provide enough document to explain benefit and cost of new metrics.
KIP-581 (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148642648)
JIRA (https://issues.apache.org/jira/browse/KAFKA-9730)
Please take a look :)
Best Regards,
Chia-Ping
Re: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so
as to observe the partition metrics on the same broker
Posted by Kamal Chandraprakash <ka...@gmail.com>.
https://issues.apache.org/jira/browse/KAFKA-7219 is identical to this one.
On Tue, Mar 24, 2020 at 10:15 AM Xue LIU <by...@gmail.com> wrote:
> Hi Chai-Ping,
>
> Thank you for bringing up this proposal. My team also has the need to have
> per partition metrics on traffic, which would be used by our partition
> rebalancer, I have a prototype running in test cluster that looks good so
> far.
>
> Some thoughts on bringing this to larger scale:
>
> 1. Simply adding another dimension (tag) to BrokerTopicsMetrics might cause
> pressure on Jmx? For our cluster with 30,000+ partitions, 300+ topics, this
> might cause a explode in number of registered Mbeans.
>
> 2. We may consider 1) Create another BrokerPartitonsMetrics 2) Opt them out
> from JmxReporter
>
> Thanks,
> Xue
>
>
>
> On Wed, Mar 18, 2020 at 10:26 PM Chia-Ping Tsai <ch...@apache.org>
> wrote:
>
> > hi Cheng
> >
> > seems the "Next KIP Number" is not synced :(
> >
> > I have updated my KIP number and corrected the "Next KIP Number"
> >
> > thanks for the reminder.
> >
> > On 2020/03/19 02:32:55, "Cheng Pan" <37...@qq.com> wrote:
> > > Hi Chia-Ping,
> > >
> > >
> > > I'm afraid the KIP number is conflict. I created (KIP-581: Value of
> > optional null field which has default value) yesterday early, and insert
> it
> > into table [KIPs under discussion].
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > >
> > >
> > > Would you please choose another KIP number, maybe 583, I also see
> > KIP-582 discussion thread appears in the mail list, and insert into table
> > [KIPs under discussion].
> > >
> > >
> > > Best Regards,
> > > Cheng Pan
> > > ------------------ Original ------------------
> > > From: "Chia-Ping Tsai";<ch...@apache.org>;
> > > Send time: Thursday, Mar 19, 2020 1:48 AM
> > > To: "dev"<de...@kafka.apache.org>;
> > >
> > > Subject: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics
> > so as to observe the partition metrics on the same broker
> > >
> > >
> > >
> > > hi
> > >
> > > this ticket is about to records partition metrics rather than topic
> > metrics. It helps us to observe more precis metrics for specify
> partition.
> > The downside is that broker needs more space to keep metrics and the
> origin
> > metrics interface (tags) is broken since this ticket adds new tag
> > "partition=xxx" to it. The alternative is an new config flag used to
> enable
> > new metrics (of course, it is false by default). Also, we ought to
> provide
> > enough document to explain benefit and cost of new metrics.
> > >
> > > KIP-581 (
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148642648
> > )
> > >
> > > JIRA (https://issues.apache.org/jira/browse/KAFKA-9730)
> > >
> > > Please take a look :)
> > >
> > > Best Regards,
> > > Chia-Ping
> >
>
Re: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so
as to observe the partition metrics on the same broker
Posted by Xue LIU <by...@gmail.com>.
Hi Chai-Ping,
Thank you for bringing up this proposal. My team also has the need to have
per partition metrics on traffic, which would be used by our partition
rebalancer, I have a prototype running in test cluster that looks good so
far.
Some thoughts on bringing this to larger scale:
1. Simply adding another dimension (tag) to BrokerTopicsMetrics might cause
pressure on Jmx? For our cluster with 30,000+ partitions, 300+ topics, this
might cause a explode in number of registered Mbeans.
2. We may consider 1) Create another BrokerPartitonsMetrics 2) Opt them out
from JmxReporter
Thanks,
Xue
On Wed, Mar 18, 2020 at 10:26 PM Chia-Ping Tsai <ch...@apache.org> wrote:
> hi Cheng
>
> seems the "Next KIP Number" is not synced :(
>
> I have updated my KIP number and corrected the "Next KIP Number"
>
> thanks for the reminder.
>
> On 2020/03/19 02:32:55, "Cheng Pan" <37...@qq.com> wrote:
> > Hi Chia-Ping,
> >
> >
> > I'm afraid the KIP number is conflict. I created (KIP-581: Value of
> optional null field which has default value) yesterday early, and insert it
> into table [KIPs under discussion].
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> >
> >
> > Would you please choose another KIP number, maybe 583, I also see
> KIP-582 discussion thread appears in the mail list, and insert into table
> [KIPs under discussion].
> >
> >
> > Best Regards,
> > Cheng Pan
> > ------------------ Original ------------------
> > From: "Chia-Ping Tsai";<ch...@apache.org>;
> > Send time: Thursday, Mar 19, 2020 1:48 AM
> > To: "dev"<de...@kafka.apache.org>;
> >
> > Subject: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics
> so as to observe the partition metrics on the same broker
> >
> >
> >
> > hi
> >
> > this ticket is about to records partition metrics rather than topic
> metrics. It helps us to observe more precis metrics for specify partition.
> The downside is that broker needs more space to keep metrics and the origin
> metrics interface (tags) is broken since this ticket adds new tag
> "partition=xxx" to it. The alternative is an new config flag used to enable
> new metrics (of course, it is false by default). Also, we ought to provide
> enough document to explain benefit and cost of new metrics.
> >
> > KIP-581 (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148642648
> )
> >
> > JIRA (https://issues.apache.org/jira/browse/KAFKA-9730)
> >
> > Please take a look :)
> >
> > Best Regards,
> > Chia-Ping
>
Re: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker
Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Cheng
seems the "Next KIP Number" is not synced :(
I have updated my KIP number and corrected the "Next KIP Number"
thanks for the reminder.
On 2020/03/19 02:32:55, "Cheng Pan" <37...@qq.com> wrote:
> Hi Chia-Ping,
>
>
> I'm afraid the KIP number is conflict. I created (KIP-581: Value of optional null field which has default value) yesterday early, and insert it into table [KIPs under discussion].
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
>
>
> Would you please choose another KIP number, maybe 583, I also see KIP-582 discussion thread appears in the mail list, and insert into table [KIPs under discussion].
>
>
> Best Regards,
> Cheng Pan
> ------------------ Original ------------------
> From: "Chia-Ping Tsai";<ch...@apache.org>;
> Send time: Thursday, Mar 19, 2020 1:48 AM
> To: "dev"<de...@kafka.apache.org>;
>
> Subject: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker
>
>
>
> hi
>
> this ticket is about to records partition metrics rather than topic metrics. It helps us to observe more precis metrics for specify partition. The downside is that broker needs more space to keep metrics and the origin metrics interface (tags) is broken since this ticket adds new tag "partition=xxx" to it. The alternative is an new config flag used to enable new metrics (of course, it is false by default). Also, we ought to provide enough document to explain benefit and cost of new metrics.
>
> KIP-581 (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148642648)
>
> JIRA (https://issues.apache.org/jira/browse/KAFKA-9730)
>
> Please take a look :)
>
> Best Regards,
> Chia-Ping
Re: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker
Posted by Cheng Pan <37...@qq.com>.
Hi Chia-Ping,
I'm afraid the KIP number is conflict. I created (KIP-581: Value of optional null field which has default value) yesterday early, and insert it into table [KIPs under discussion].
https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
Would you please choose another KIP number, maybe 583, I also see KIP-582 discussion thread appears in the mail list, and insert into table [KIPs under discussion].
Best Regards,
Cheng Pan
------------------ Original ------------------
From: "Chia-Ping Tsai";<ch...@apache.org>;
Send time: Thursday, Mar 19, 2020 1:48 AM
To: "dev"<de...@kafka.apache.org>;
Subject: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker
hi
this ticket is about to records partition metrics rather than topic metrics. It helps us to observe more precis metrics for specify partition. The downside is that broker needs more space to keep metrics and the origin metrics interface (tags) is broken since this ticket adds new tag "partition=xxx" to it. The alternative is an new config flag used to enable new metrics (of course, it is false by default). Also, we ought to provide enough document to explain benefit and cost of new metrics.
KIP-581 (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148642648)
JIRA (https://issues.apache.org/jira/browse/KAFKA-9730)
Please take a look :)
Best Regards,
Chia-Ping