You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Dong Lin <li...@gmail.com> on 2017/05/28 08:05:31 UTC
[DISCUSS] KIP-164 Add unavailablePartitionCount and per-partition
Unavailable metrics
Hi,
We created KIP-164 to propose adding per-partition metric *Unavailable* and
per-broker metric *UnavailablePartitionCount*
The KIP wik can be found at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+unavailablePartitionCount+and+per-partition+Unavailable+metrics
.
Comments are welcome.
Thanks,
Dong
Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and per-partition
Unavailable metrics
Posted by Dong Lin <li...@gmail.com>.
Hey Ismael,
Thanks for the feedback! Could you please vote for the KIP if it looks
good? Then I will find two more committers to vote as well.
Thanks,
Dong
On Tue, May 30, 2017 at 9:08 AM, Dong Lin <li...@gmail.com> wrote:
> Thanks Edoardo and everyone for the comment! That is a very good point. I
> have updated to wiki to use UnderMinIsrPartitionCount as the per-broker
> metric name and UnderMinIsr as the per-partition metric name. The
> motivation section has also been updated to clarify how the existence of
> UnderMinIsrPartition reduces the availability of the Kafka service.
>
> Please find the latest wiki at https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-164-+Add+UnderMinIsrPartitionCount+and+
> per-partition+UnderMinIsr+metrics .
>
>
>
> On Tue, May 30, 2017 at 5:35 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
>> Thanks for the KIP, Dong. I agree that that the metrics are useful. Like
>> Edoardo and Mickael said, it seems like it may be better to choose a
>> different name. A couple of additional suggestions:
>> `UnderMinIsrPartitionCount` and `UnderMinIsr`.
>>
>> Ismael
>>
>> On Tue, May 30, 2017 at 12:04 PM, Mickael Maison <
>> mickael.maison@gmail.com>
>> wrote:
>>
>> > What about simply calling them 'BelowIsrPartitionCount' and 'BelowIsr' ?
>> >
>> > On Tue, May 30, 2017 at 11:40 AM, Edoardo Comar <EC...@uk.ibm.com>
>> wrote:
>> > > Hi Dong,
>> > >
>> > > many thanks for the KIP. It's a very useful metric.
>> > >
>> > > by saying
>> > >> Unavailable partitions could be most easily defined as “The number of
>> > > partitions that this broker leads for which the ISR is insufficient to
>> > > meet the minimum ISR required.”
>> > >
>> > > I presume you meant to call 'Unavailable' the partitions whose
>> ISR.size <
>> > > min.insync ?
>> > >
>> > > Now, a partition whose ISR is < min.insync can be still used to
>> consume
>> > > messages from. It also can be used to produce messages to, as long as
>> the
>> > > producer does not request acks=-1 (i.e. acks=all).
>> > >
>> > > So it is not exactly 'Unavailable' ... perhaps we could call it
>> 'Unsafe'
>> > ?
>> > > Or the community can come up with a better name.
>> > >
>> > > I recently had a few discussions about the issue, and I opened a PR to
>> > > update the docs (that's still hoping to be reviewed and merged ...
>> hint
>> > > hint :-)
>> > > https://github.com/apache/kafka/pull/3035
>> > > https://issues.apache.org/jira/browse/KAFKA-5290
>> > >
>> > > Thanks!
>> > > Edo
>> > > --------------------------------------------------
>> > > Edoardo Comar
>> > > IBM Message Hub
>> > > ecomar@uk.ibm.com
>> > > IBM UK Ltd, Hursley Park, SO21 2JN
>> > >
>> > >
>> > >
>> > >
>> > > From: Mickael Maison <mi...@gmail.com>
>> > > To: dev@kafka.apache.org
>> > > Date: 30/05/2017 10:51
>> > > Subject: Re: [DISCUSS] KIP-164 Add unavailablePartitionCount
>> and
>> > > per-partition Unavailable metrics
>> > >
>> > >
>> > >
>> > > +1
>> > > It's a mystery how this didn't already exist as it's one of the key
>> > > cluster's health indicator
>> > >
>> > > On Mon, May 29, 2017 at 9:18 AM, Gwen Shapira <gw...@confluent.io>
>> wrote:
>> > >> Hi,
>> > >>
>> > >> Sounds good. I was sure this existed already for some reason :)
>> > >>
>> > >> On Sun, May 28, 2017 at 11:06 AM Dong Lin <li...@gmail.com>
>> wrote:
>> > >>
>> > >>> Hi,
>> > >>>
>> > >>> We created KIP-164 to propose adding per-partition metric
>> *Unavailable*
>> > > and
>> > >>> per-broker metric *UnavailablePartitionCount*
>> > >>>
>> > >>> The KIP wik can be found at
>> > >>>
>> > >>>
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+
>> > unavailablePartitionCount+and+per-partition+Unavailable+metrics
>> > >
>> > >>> .
>> > >>>
>> > >>> Comments are welcome.
>> > >>>
>> > >>> Thanks,
>> > >>> Dong
>> > >>>
>> > >
>> > >
>> > >
>> > >
>> > > Unless stated otherwise above:
>> > > IBM United Kingdom Limited - Registered in England and Wales with
>> number
>> > > 741598.
>> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>> > 3AU
>> > >
>> >
>>
>
>
Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and per-partition
Unavailable metrics
Posted by Dong Lin <li...@gmail.com>.
Thanks Edoardo and everyone for the comment! That is a very good point. I
have updated to wiki to use UnderMinIsrPartitionCount as the per-broker
metric name and UnderMinIsr as the per-partition metric name. The
motivation section has also been updated to clarify how the existence of
UnderMinIsrPartition reduces the availability of the Kafka service.
Please find the latest wiki at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+UnderMinIsrPartitionCount+and+per-partition+UnderMinIsr+metrics
.
On Tue, May 30, 2017 at 5:35 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Thanks for the KIP, Dong. I agree that that the metrics are useful. Like
> Edoardo and Mickael said, it seems like it may be better to choose a
> different name. A couple of additional suggestions:
> `UnderMinIsrPartitionCount` and `UnderMinIsr`.
>
> Ismael
>
> On Tue, May 30, 2017 at 12:04 PM, Mickael Maison <mickael.maison@gmail.com
> >
> wrote:
>
> > What about simply calling them 'BelowIsrPartitionCount' and 'BelowIsr' ?
> >
> > On Tue, May 30, 2017 at 11:40 AM, Edoardo Comar <EC...@uk.ibm.com>
> wrote:
> > > Hi Dong,
> > >
> > > many thanks for the KIP. It's a very useful metric.
> > >
> > > by saying
> > >> Unavailable partitions could be most easily defined as “The number of
> > > partitions that this broker leads for which the ISR is insufficient to
> > > meet the minimum ISR required.”
> > >
> > > I presume you meant to call 'Unavailable' the partitions whose
> ISR.size <
> > > min.insync ?
> > >
> > > Now, a partition whose ISR is < min.insync can be still used to consume
> > > messages from. It also can be used to produce messages to, as long as
> the
> > > producer does not request acks=-1 (i.e. acks=all).
> > >
> > > So it is not exactly 'Unavailable' ... perhaps we could call it
> 'Unsafe'
> > ?
> > > Or the community can come up with a better name.
> > >
> > > I recently had a few discussions about the issue, and I opened a PR to
> > > update the docs (that's still hoping to be reviewed and merged ... hint
> > > hint :-)
> > > https://github.com/apache/kafka/pull/3035
> > > https://issues.apache.org/jira/browse/KAFKA-5290
> > >
> > > Thanks!
> > > Edo
> > > --------------------------------------------------
> > > Edoardo Comar
> > > IBM Message Hub
> > > ecomar@uk.ibm.com
> > > IBM UK Ltd, Hursley Park, SO21 2JN
> > >
> > >
> > >
> > >
> > > From: Mickael Maison <mi...@gmail.com>
> > > To: dev@kafka.apache.org
> > > Date: 30/05/2017 10:51
> > > Subject: Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and
> > > per-partition Unavailable metrics
> > >
> > >
> > >
> > > +1
> > > It's a mystery how this didn't already exist as it's one of the key
> > > cluster's health indicator
> > >
> > > On Mon, May 29, 2017 at 9:18 AM, Gwen Shapira <gw...@confluent.io>
> wrote:
> > >> Hi,
> > >>
> > >> Sounds good. I was sure this existed already for some reason :)
> > >>
> > >> On Sun, May 28, 2017 at 11:06 AM Dong Lin <li...@gmail.com>
> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> We created KIP-164 to propose adding per-partition metric
> *Unavailable*
> > > and
> > >>> per-broker metric *UnavailablePartitionCount*
> > >>>
> > >>> The KIP wik can be found at
> > >>>
> > >>>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+
> > unavailablePartitionCount+and+per-partition+Unavailable+metrics
> > >
> > >>> .
> > >>>
> > >>> Comments are welcome.
> > >>>
> > >>> Thanks,
> > >>> Dong
> > >>>
> > >
> > >
> > >
> > >
> > > Unless stated otherwise above:
> > > IBM United Kingdom Limited - Registered in England and Wales with
> number
> > > 741598.
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> > 3AU
> > >
> >
>
Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and per-partition
Unavailable metrics
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks for the KIP, Dong. I agree that that the metrics are useful. Like
Edoardo and Mickael said, it seems like it may be better to choose a
different name. A couple of additional suggestions:
`UnderMinIsrPartitionCount` and `UnderMinIsr`.
Ismael
On Tue, May 30, 2017 at 12:04 PM, Mickael Maison <mi...@gmail.com>
wrote:
> What about simply calling them 'BelowIsrPartitionCount' and 'BelowIsr' ?
>
> On Tue, May 30, 2017 at 11:40 AM, Edoardo Comar <EC...@uk.ibm.com> wrote:
> > Hi Dong,
> >
> > many thanks for the KIP. It's a very useful metric.
> >
> > by saying
> >> Unavailable partitions could be most easily defined as “The number of
> > partitions that this broker leads for which the ISR is insufficient to
> > meet the minimum ISR required.”
> >
> > I presume you meant to call 'Unavailable' the partitions whose ISR.size <
> > min.insync ?
> >
> > Now, a partition whose ISR is < min.insync can be still used to consume
> > messages from. It also can be used to produce messages to, as long as the
> > producer does not request acks=-1 (i.e. acks=all).
> >
> > So it is not exactly 'Unavailable' ... perhaps we could call it 'Unsafe'
> ?
> > Or the community can come up with a better name.
> >
> > I recently had a few discussions about the issue, and I opened a PR to
> > update the docs (that's still hoping to be reviewed and merged ... hint
> > hint :-)
> > https://github.com/apache/kafka/pull/3035
> > https://issues.apache.org/jira/browse/KAFKA-5290
> >
> > Thanks!
> > Edo
> > --------------------------------------------------
> > Edoardo Comar
> > IBM Message Hub
> > ecomar@uk.ibm.com
> > IBM UK Ltd, Hursley Park, SO21 2JN
> >
> >
> >
> >
> > From: Mickael Maison <mi...@gmail.com>
> > To: dev@kafka.apache.org
> > Date: 30/05/2017 10:51
> > Subject: Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and
> > per-partition Unavailable metrics
> >
> >
> >
> > +1
> > It's a mystery how this didn't already exist as it's one of the key
> > cluster's health indicator
> >
> > On Mon, May 29, 2017 at 9:18 AM, Gwen Shapira <gw...@confluent.io> wrote:
> >> Hi,
> >>
> >> Sounds good. I was sure this existed already for some reason :)
> >>
> >> On Sun, May 28, 2017 at 11:06 AM Dong Lin <li...@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> We created KIP-164 to propose adding per-partition metric *Unavailable*
> > and
> >>> per-broker metric *UnavailablePartitionCount*
> >>>
> >>> The KIP wik can be found at
> >>>
> >>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+
> unavailablePartitionCount+and+per-partition+Unavailable+metrics
> >
> >>> .
> >>>
> >>> Comments are welcome.
> >>>
> >>> Thanks,
> >>> Dong
> >>>
> >
> >
> >
> >
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
> >
>
Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and per-partition
Unavailable metrics
Posted by Mickael Maison <mi...@gmail.com>.
What about simply calling them 'BelowIsrPartitionCount' and 'BelowIsr' ?
On Tue, May 30, 2017 at 11:40 AM, Edoardo Comar <EC...@uk.ibm.com> wrote:
> Hi Dong,
>
> many thanks for the KIP. It's a very useful metric.
>
> by saying
>> Unavailable partitions could be most easily defined as “The number of
> partitions that this broker leads for which the ISR is insufficient to
> meet the minimum ISR required.”
>
> I presume you meant to call 'Unavailable' the partitions whose ISR.size <
> min.insync ?
>
> Now, a partition whose ISR is < min.insync can be still used to consume
> messages from. It also can be used to produce messages to, as long as the
> producer does not request acks=-1 (i.e. acks=all).
>
> So it is not exactly 'Unavailable' ... perhaps we could call it 'Unsafe' ?
> Or the community can come up with a better name.
>
> I recently had a few discussions about the issue, and I opened a PR to
> update the docs (that's still hoping to be reviewed and merged ... hint
> hint :-)
> https://github.com/apache/kafka/pull/3035
> https://issues.apache.org/jira/browse/KAFKA-5290
>
> Thanks!
> Edo
> --------------------------------------------------
> Edoardo Comar
> IBM Message Hub
> ecomar@uk.ibm.com
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
>
> From: Mickael Maison <mi...@gmail.com>
> To: dev@kafka.apache.org
> Date: 30/05/2017 10:51
> Subject: Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and
> per-partition Unavailable metrics
>
>
>
> +1
> It's a mystery how this didn't already exist as it's one of the key
> cluster's health indicator
>
> On Mon, May 29, 2017 at 9:18 AM, Gwen Shapira <gw...@confluent.io> wrote:
>> Hi,
>>
>> Sounds good. I was sure this existed already for some reason :)
>>
>> On Sun, May 28, 2017 at 11:06 AM Dong Lin <li...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We created KIP-164 to propose adding per-partition metric *Unavailable*
> and
>>> per-broker metric *UnavailablePartitionCount*
>>>
>>> The KIP wik can be found at
>>>
>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+unavailablePartitionCount+and+per-partition+Unavailable+metrics
>
>>> .
>>>
>>> Comments are welcome.
>>>
>>> Thanks,
>>> Dong
>>>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and per-partition
Unavailable metrics
Posted by Edoardo Comar <EC...@uk.ibm.com>.
Hi Dong,
many thanks for the KIP. It's a very useful metric.
by saying
> Unavailable partitions could be most easily defined as “The number of
partitions that this broker leads for which the ISR is insufficient to
meet the minimum ISR required.”
I presume you meant to call 'Unavailable' the partitions whose ISR.size <
min.insync ?
Now, a partition whose ISR is < min.insync can be still used to consume
messages from. It also can be used to produce messages to, as long as the
producer does not request acks=-1 (i.e. acks=all).
So it is not exactly 'Unavailable' ... perhaps we could call it 'Unsafe' ?
Or the community can come up with a better name.
I recently had a few discussions about the issue, and I opened a PR to
update the docs (that's still hoping to be reviewed and merged ... hint
hint :-)
https://github.com/apache/kafka/pull/3035
https://issues.apache.org/jira/browse/KAFKA-5290
Thanks!
Edo
--------------------------------------------------
Edoardo Comar
IBM Message Hub
ecomar@uk.ibm.com
IBM UK Ltd, Hursley Park, SO21 2JN
From: Mickael Maison <mi...@gmail.com>
To: dev@kafka.apache.org
Date: 30/05/2017 10:51
Subject: Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and
per-partition Unavailable metrics
+1
It's a mystery how this didn't already exist as it's one of the key
cluster's health indicator
On Mon, May 29, 2017 at 9:18 AM, Gwen Shapira <gw...@confluent.io> wrote:
> Hi,
>
> Sounds good. I was sure this existed already for some reason :)
>
> On Sun, May 28, 2017 at 11:06 AM Dong Lin <li...@gmail.com> wrote:
>
>> Hi,
>>
>> We created KIP-164 to propose adding per-partition metric *Unavailable*
and
>> per-broker metric *UnavailablePartitionCount*
>>
>> The KIP wik can be found at
>>
>>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+unavailablePartitionCount+and+per-partition+Unavailable+metrics
>> .
>>
>> Comments are welcome.
>>
>> Thanks,
>> Dong
>>
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and per-partition
Unavailable metrics
Posted by Mickael Maison <mi...@gmail.com>.
+1
It's a mystery how this didn't already exist as it's one of the key
cluster's health indicator
On Mon, May 29, 2017 at 9:18 AM, Gwen Shapira <gw...@confluent.io> wrote:
> Hi,
>
> Sounds good. I was sure this existed already for some reason :)
>
> On Sun, May 28, 2017 at 11:06 AM Dong Lin <li...@gmail.com> wrote:
>
>> Hi,
>>
>> We created KIP-164 to propose adding per-partition metric *Unavailable* and
>> per-broker metric *UnavailablePartitionCount*
>>
>> The KIP wik can be found at
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+unavailablePartitionCount+and+per-partition+Unavailable+metrics
>> .
>>
>> Comments are welcome.
>>
>> Thanks,
>> Dong
>>
Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and per-partition
Unavailable metrics
Posted by Gwen Shapira <gw...@confluent.io>.
Hi,
Sounds good. I was sure this existed already for some reason :)
On Sun, May 28, 2017 at 11:06 AM Dong Lin <li...@gmail.com> wrote:
> Hi,
>
> We created KIP-164 to propose adding per-partition metric *Unavailable* and
> per-broker metric *UnavailablePartitionCount*
>
> The KIP wik can be found at
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+unavailablePartitionCount+and+per-partition+Unavailable+metrics
> .
>
> Comments are welcome.
>
> Thanks,
> Dong
>