You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Mickael Maison <mi...@gmail.com> on 2019/11/13 12:17:51 UTC

[VOTE] KIP-409: Allow creating under-replicated topics and partitions

Hi all,

I'd like to start a vote on KIP-409: Allow creating under-replicated
topics and partitions
https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions

Thanks

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Mickael Maison <mi...@gmail.com>.

Bumping this thread.
We are still missing binding votes.

Please take a look at the KIP
https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
and let me know if you have any feedback.
Thanks

On Fri, Mar 6, 2020 at 10:03 AM Mickael Maison <mi...@gmail.com> wrote:
>
> Bumping this thread once again.
> So far we only have 4 non binding votes.
>
> Please take a look at the KIP, share any feedback and consider voting.
> Thanks
>
> On Thu, Feb 27, 2020 at 12:03 AM Ryanne Dolan <ry...@gmail.com> wrote:
> >
> > Hey all, please consider voting for this KIP.  It's really a shame that
> > topic creation is impossible when clusters are under-provisioned, which is
> > not uncommon in a dynamic environment like Kubernetes.
> >
> > Ryanne
> >
> > On Thu, Feb 6, 2020 at 10:57 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> >
> > > I have not seen any new feedback nor votes.
> > > Bumping this thread again
> > >
> > > On Mon, Jan 27, 2020 at 3:55 PM Mickael Maison <mi...@gmail.com>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > We are now at 4 non-binding votes but still no binding votes.
> > > > I have not seen any outstanding questions in the DISCUSS thread. If
> > > > you have any feedback, please let me know.
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Thu, Jan 16, 2020 at 2:03 PM M. Manna <ma...@gmail.com> wrote:
> > > > >
> > > > > MIckael,
> > > > >
> > > > >
> > > > >
> > > > > On Thu, 16 Jan 2020 at 14:01, Mickael Maison <mickael.maison@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Manna,
> > > > > >
> > > > > > In your example, the topic 'dummy' is not under replicated. It just
> > > > > > has 1 replica. A topic under replicated is a topic with less ISRs
> > > than
> > > > > > replicas.
> > > > > >
> > > > > > Having under replicated topics is relatively common in a Kafka
> > > > > > cluster, it happens everytime is broker is down. However Kafka does
> > > > > > not permit it to happen at topic creation. Currently at creation,
> > > > > > Kafka requires to have at least as many brokers as the replication
> > > > > > factor. This KIP addresses this limitation.
> > > > > >
> > > > > > Regarding your 2nd point. When rack awareness is enabled, Kafka tries
> > > > > > to distribute partitions across racks. When all brokers in a rack are
> > > > > > down (ie: a zone is offline), you can end up with partitions not well
> > > > > > distributed even with rack awareness. There are currently no easy way
> > > > > > to track such partitions so I decided to not attempt addressing this
> > > > > > issue in this KIP.
> > > > > >
> > > > > > I hope that answers your questions.
> > > > > >
> > > > >
> > > > >  It does and I appreciate you taking time and explaining this.
> > > > >
> > > > >  +1 (binding) if I haven't already.
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
> > > > > > <ka...@gmail.com> wrote:
> > > > > > >
> > > > > > > +1 (non-binding). Thanks for the KIP!
> > > > > > >
> > > > > > > On Mon, Jan 13, 2020 at 1:58 PM M. Manna <ma...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hi Mikael,
> > > > > > > >
> > > > > > > > Apologies for last minute question, as I just caught up with it.
> > > > > > Thanks for
> > > > > > > > your work on the KIP.
> > > > > > > >
> > > > > > > > Just trying to get your thoughts on one thing (I might have
> > > > > > misunderstood
> > > > > > > > it) - currently it's possible (even though I am strongly against
> > > it) to
> > > > > > > > create Kafka topics which are under-replicated; despite all
> > > brokers
> > > > > > being
> > > > > > > > online. This the the output of an intentionally under-replicated
> > > topic
> > > > > > > > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> > > > > > > >
> > > > > > > >
> > > > > > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$
> > > ./kafka-topics.sh
> > > > > > > > --create --topic dummy --partitions 6 --replication-factor 1
> > > > > > > > --bootstrap-server localhost:9092
> > > > > > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$
> > > ./kafka-topics.sh
> > > > > > > > --describe --topic dummy  --bootstrap-server localhost:9092
> > > > > > > > Topic:dummy     PartitionCount:6        ReplicationFactor:1
> > > > > > > >
> > > > > > > >
> > > > > >
> > > Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> > > > > > > >         Topic: dummy    Partition: 0    Leader: 3
> > >  Replicas: 3
> > > > > > > > Isr: 3
> > > > > > > >         Topic: dummy    Partition: 1    Leader: 1
> > >  Replicas: 1
> > > > > > > > Isr: 1
> > > > > > > >         Topic: dummy    Partition: 2    Leader: 2
> > >  Replicas: 2
> > > > > > > > Isr: 2
> > > > > > > >         Topic: dummy    Partition: 3    Leader: 3
> > >  Replicas: 3
> > > > > > > > Isr: 3
> > > > > > > >         Topic: dummy    Partition: 4    Leader: 1
> > >  Replicas: 1
> > > > > > > > Isr: 1
> > > > > > > >         Topic: dummy    Partition: 5    Leader: 2
> > >  Replicas: 2
> > > > > > > > Isr: 2
> > > > > > > >
> > > > > > > >  This is with respect to the following statement on your KIP
> > > (i.e.
> > > > > > > > under-replicated topic creation is also permitted when none is
> > > > > > offline):
> > > > > > > >
> > > > > > > > *but note that this may already happen (without this KIP) when
> > > > > > > > > topics/partitions are created while all brokers in a rack are
> > > offline
> > > > > > > > (ie:
> > > > > > > > > an availability zone is offline). Tracking topics/partitions
> > > not
> > > > > > > > optimally
> > > > > > > > > spread across all racks can be tackled in a follow up KIP.  *
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Did you mean to say that such under-replicated topics (including
> > > > > > > > human-created ones) will be handled in a separete KIP ?
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 13 Jan 2020 at 10:15, Mickael Maison <
> > > mickael.maison@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all.
> > > > > > > > >
> > > > > > > > > With 2.5.0 approaching, bumping this thread once more as
> > > feedback or
> > > > > > > > > votes would be nice.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <
> > > tbentley@redhat.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1 non-binding. Thanks!
> > > > > > > > > >
> > > > > > > > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Mickael,
> > > > > > > > > > >
> > > > > > > > > > > thanks for your response! That all makes perfect sense and
> > > I
> > > > > > cannot
> > > > > > > > > > > give any actual use cases for where what I asked about
> > > would be
> > > > > > > > useful
> > > > > > > > > > > :)
> > > > > > > > > > > It was more the idle thought if this might be low hanging
> > > fruit
> > > > > > while
> > > > > > > > > > > changing this anyway to avoid having to circle back later
> > > on and
> > > > > > > > > > > wanted to at least mention it.
> > > > > > > > > > >
> > > > > > > > > > > I am totally happy either way!
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Sönke
> > > > > > > > > > >
> > > > > > > > > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> > > > > > > > mickael.maison@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks Sönke for the feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > I debated this point quite a bit before deciding to base
> > > > > > creation
> > > > > > > > > > > > around "min.insync.replicas".
> > > > > > > > > > > >
> > > > > > > > > > > > For me, the goal of this KIP is to enable administrators
> > > to
> > > > > > provide
> > > > > > > > > > > > higher availability. In a 3 node cluster configured for
> > > high
> > > > > > > > > > > > availability (3 replicas, 2 min ISR), by enabling this
> > > feature,
> > > > > > > > > > > > clusters should be fully usable even when 1 broker is
> > > down.
> > > > > > This
> > > > > > > > > > > > should cover all "normal" maintenance operations like a
> > > rolling
> > > > > > > > > > > > restart or just the recovery of a broker.
> > > > > > > > > > > >
> > > > > > > > > > > > At the moment, when creating a topic/partition, the
> > > assumption
> > > > > > is
> > > > > > > > > that
> > > > > > > > > > > > the resource will be fully functioning. This KIP does not
> > > > > > change
> > > > > > > > this
> > > > > > > > > > > > assumption. If this is something someone wants, I think
> > > it
> > > > > > should
> > > > > > > > be
> > > > > > > > > > > > handled in a different KIP that targets that use case. By
> > > > > > relying
> > > > > > > > on
> > > > > > > > > > > > "min.insync.replicas", we don't break any assumptions
> > > the user
> > > > > > has
> > > > > > > > > and
> > > > > > > > > > > > this should be fully transparent from the user point of
> > > view.
> > > > > > > > > > > >
> > > > > > > > > > > > About "min.insync.replicas", one caveat that is not
> > > explicit
> > > > > > in the
> > > > > > > > > > > > KIP is that it's currently possible to create topics
> > > with less
> > > > > > > > > > > > replicas than this settings. For that reason, I think the
> > > > > > > > > > > > implementation will actually rely on min(replicas,
> > > min-isr)
> > > > > > instead
> > > > > > > > > of
> > > > > > > > > > > > simply min.insync.replicas. I have updated the KIP to
> > > > > > explicitly
> > > > > > > > > > > > mention this point.
> > > > > > > > > > > >
> > > > > > > > > > > > I hope that answers your question, let me know.
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > > > > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Michael,
> > > > > > > > > > > > >
> > > > > > > > > > > > > that sounds like a useful addition! I can't help but
> > > wonder
> > > > > > > > > whether by
> > > > > > > > > > > > > leaving in the restriction that "min.insync.replicas"
> > > has to
> > > > > > be
> > > > > > > > > > > > > satisfied we'll be back here in a years time because
> > > someone
> > > > > > has
> > > > > > > > a
> > > > > > > > > > > > > scenario where he or she wants to go below that :)
> > > > > > > > > > > > > I don't have a strong opinion either way to be honest,
> > > just a
> > > > > > > > > random
> > > > > > > > > > > > > thought when reading the KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Sönke
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <
> > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 non-binding, thx
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Bumping this thread, I've not seen any votes or
> > > feedback.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > > > > > > > > > > <mi...@gmail.com> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'd like to start a vote on KIP-409: Allow
> > > creating
> > > > > > > > > > > under-replicated
> > > > > > > > > > > > > > > > topics and partitions
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Sönke Liebau
> > > > > > > > > > > > > Partner
> > > > > > > > > > > > > Tel. +49 179 7940878
> > > > > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880
> > > Wedel -
> > > > > > > > > Germany
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Sönke Liebau
> > > > > > > > > > > Partner
> > > > > > > > > > > Tel. +49 179 7940878
> > > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880
> > > Wedel -
> > > > > > Germany
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > >

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Mickael Maison <mi...@gmail.com>.

Bumping this thread once again.
So far we only have 4 non binding votes.

Please take a look at the KIP, share any feedback and consider voting.
Thanks

On Thu, Feb 27, 2020 at 12:03 AM Ryanne Dolan <ry...@gmail.com> wrote:
>
> Hey all, please consider voting for this KIP.  It's really a shame that
> topic creation is impossible when clusters are under-provisioned, which is
> not uncommon in a dynamic environment like Kubernetes.
>
> Ryanne
>
> On Thu, Feb 6, 2020 at 10:57 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > I have not seen any new feedback nor votes.
> > Bumping this thread again
> >
> > On Mon, Jan 27, 2020 at 3:55 PM Mickael Maison <mi...@gmail.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > We are now at 4 non-binding votes but still no binding votes.
> > > I have not seen any outstanding questions in the DISCUSS thread. If
> > > you have any feedback, please let me know.
> > >
> > > Thanks
> > >
> > >
> > > On Thu, Jan 16, 2020 at 2:03 PM M. Manna <ma...@gmail.com> wrote:
> > > >
> > > > MIckael,
> > > >
> > > >
> > > >
> > > > On Thu, 16 Jan 2020 at 14:01, Mickael Maison <mickael.maison@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Manna,
> > > > >
> > > > > In your example, the topic 'dummy' is not under replicated. It just
> > > > > has 1 replica. A topic under replicated is a topic with less ISRs
> > than
> > > > > replicas.
> > > > >
> > > > > Having under replicated topics is relatively common in a Kafka
> > > > > cluster, it happens everytime is broker is down. However Kafka does
> > > > > not permit it to happen at topic creation. Currently at creation,
> > > > > Kafka requires to have at least as many brokers as the replication
> > > > > factor. This KIP addresses this limitation.
> > > > >
> > > > > Regarding your 2nd point. When rack awareness is enabled, Kafka tries
> > > > > to distribute partitions across racks. When all brokers in a rack are
> > > > > down (ie: a zone is offline), you can end up with partitions not well
> > > > > distributed even with rack awareness. There are currently no easy way
> > > > > to track such partitions so I decided to not attempt addressing this
> > > > > issue in this KIP.
> > > > >
> > > > > I hope that answers your questions.
> > > > >
> > > >
> > > >  It does and I appreciate you taking time and explaining this.
> > > >
> > > >  +1 (binding) if I haven't already.
> > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
> > > > > <ka...@gmail.com> wrote:
> > > > > >
> > > > > > +1 (non-binding). Thanks for the KIP!
> > > > > >
> > > > > > On Mon, Jan 13, 2020 at 1:58 PM M. Manna <ma...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Mikael,
> > > > > > >
> > > > > > > Apologies for last minute question, as I just caught up with it.
> > > > > Thanks for
> > > > > > > your work on the KIP.
> > > > > > >
> > > > > > > Just trying to get your thoughts on one thing (I might have
> > > > > misunderstood
> > > > > > > it) - currently it's possible (even though I am strongly against
> > it) to
> > > > > > > create Kafka topics which are under-replicated; despite all
> > brokers
> > > > > being
> > > > > > > online. This the the output of an intentionally under-replicated
> > topic
> > > > > > > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> > > > > > >
> > > > > > >
> > > > > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$
> > ./kafka-topics.sh
> > > > > > > --create --topic dummy --partitions 6 --replication-factor 1
> > > > > > > --bootstrap-server localhost:9092
> > > > > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$
> > ./kafka-topics.sh
> > > > > > > --describe --topic dummy  --bootstrap-server localhost:9092
> > > > > > > Topic:dummy     PartitionCount:6        ReplicationFactor:1
> > > > > > >
> > > > > > >
> > > > >
> > Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> > > > > > >         Topic: dummy    Partition: 0    Leader: 3
> >  Replicas: 3
> > > > > > > Isr: 3
> > > > > > >         Topic: dummy    Partition: 1    Leader: 1
> >  Replicas: 1
> > > > > > > Isr: 1
> > > > > > >         Topic: dummy    Partition: 2    Leader: 2
> >  Replicas: 2
> > > > > > > Isr: 2
> > > > > > >         Topic: dummy    Partition: 3    Leader: 3
> >  Replicas: 3
> > > > > > > Isr: 3
> > > > > > >         Topic: dummy    Partition: 4    Leader: 1
> >  Replicas: 1
> > > > > > > Isr: 1
> > > > > > >         Topic: dummy    Partition: 5    Leader: 2
> >  Replicas: 2
> > > > > > > Isr: 2
> > > > > > >
> > > > > > >  This is with respect to the following statement on your KIP
> > (i.e.
> > > > > > > under-replicated topic creation is also permitted when none is
> > > > > offline):
> > > > > > >
> > > > > > > *but note that this may already happen (without this KIP) when
> > > > > > > > topics/partitions are created while all brokers in a rack are
> > offline
> > > > > > > (ie:
> > > > > > > > an availability zone is offline). Tracking topics/partitions
> > not
> > > > > > > optimally
> > > > > > > > spread across all racks can be tackled in a follow up KIP.  *
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Did you mean to say that such under-replicated topics (including
> > > > > > > human-created ones) will be handled in a separete KIP ?
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 13 Jan 2020 at 10:15, Mickael Maison <
> > mickael.maison@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all.
> > > > > > > >
> > > > > > > > With 2.5.0 approaching, bumping this thread once more as
> > feedback or
> > > > > > > > votes would be nice.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <
> > tbentley@redhat.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > +1 non-binding. Thanks!
> > > > > > > > >
> > > > > > > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Mickael,
> > > > > > > > > >
> > > > > > > > > > thanks for your response! That all makes perfect sense and
> > I
> > > > > cannot
> > > > > > > > > > give any actual use cases for where what I asked about
> > would be
> > > > > > > useful
> > > > > > > > > > :)
> > > > > > > > > > It was more the idle thought if this might be low hanging
> > fruit
> > > > > while
> > > > > > > > > > changing this anyway to avoid having to circle back later
> > on and
> > > > > > > > > > wanted to at least mention it.
> > > > > > > > > >
> > > > > > > > > > I am totally happy either way!
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Sönke
> > > > > > > > > >
> > > > > > > > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> > > > > > > mickael.maison@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Thanks Sönke for the feedback.
> > > > > > > > > > >
> > > > > > > > > > > I debated this point quite a bit before deciding to base
> > > > > creation
> > > > > > > > > > > around "min.insync.replicas".
> > > > > > > > > > >
> > > > > > > > > > > For me, the goal of this KIP is to enable administrators
> > to
> > > > > provide
> > > > > > > > > > > higher availability. In a 3 node cluster configured for
> > high
> > > > > > > > > > > availability (3 replicas, 2 min ISR), by enabling this
> > feature,
> > > > > > > > > > > clusters should be fully usable even when 1 broker is
> > down.
> > > > > This
> > > > > > > > > > > should cover all "normal" maintenance operations like a
> > rolling
> > > > > > > > > > > restart or just the recovery of a broker.
> > > > > > > > > > >
> > > > > > > > > > > At the moment, when creating a topic/partition, the
> > assumption
> > > > > is
> > > > > > > > that
> > > > > > > > > > > the resource will be fully functioning. This KIP does not
> > > > > change
> > > > > > > this
> > > > > > > > > > > assumption. If this is something someone wants, I think
> > it
> > > > > should
> > > > > > > be
> > > > > > > > > > > handled in a different KIP that targets that use case. By
> > > > > relying
> > > > > > > on
> > > > > > > > > > > "min.insync.replicas", we don't break any assumptions
> > the user
> > > > > has
> > > > > > > > and
> > > > > > > > > > > this should be fully transparent from the user point of
> > view.
> > > > > > > > > > >
> > > > > > > > > > > About "min.insync.replicas", one caveat that is not
> > explicit
> > > > > in the
> > > > > > > > > > > KIP is that it's currently possible to create topics
> > with less
> > > > > > > > > > > replicas than this settings. For that reason, I think the
> > > > > > > > > > > implementation will actually rely on min(replicas,
> > min-isr)
> > > > > instead
> > > > > > > > of
> > > > > > > > > > > simply min.insync.replicas. I have updated the KIP to
> > > > > explicitly
> > > > > > > > > > > mention this point.
> > > > > > > > > > >
> > > > > > > > > > > I hope that answers your question, let me know.
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > > > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Michael,
> > > > > > > > > > > >
> > > > > > > > > > > > that sounds like a useful addition! I can't help but
> > wonder
> > > > > > > > whether by
> > > > > > > > > > > > leaving in the restriction that "min.insync.replicas"
> > has to
> > > > > be
> > > > > > > > > > > > satisfied we'll be back here in a years time because
> > someone
> > > > > has
> > > > > > > a
> > > > > > > > > > > > scenario where he or she wants to go below that :)
> > > > > > > > > > > > I don't have a strong opinion either way to be honest,
> > just a
> > > > > > > > random
> > > > > > > > > > > > thought when reading the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Sönke
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <
> > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > +1 non-binding, thx
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Bumping this thread, I've not seen any votes or
> > feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > > > > > > > > > <mi...@gmail.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'd like to start a vote on KIP-409: Allow
> > creating
> > > > > > > > > > under-replicated
> > > > > > > > > > > > > > > topics and partitions
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Sönke Liebau
> > > > > > > > > > > > Partner
> > > > > > > > > > > > Tel. +49 179 7940878
> > > > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880
> > Wedel -
> > > > > > > > Germany
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Sönke Liebau
> > > > > > > > > > Partner
> > > > > > > > > > Tel. +49 179 7940878
> > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880
> > Wedel -
> > > > > Germany
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> >

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Ryanne Dolan <ry...@gmail.com>.

Hey all, please consider voting for this KIP.  It's really a shame that
topic creation is impossible when clusters are under-provisioned, which is
not uncommon in a dynamic environment like Kubernetes.

Ryanne

On Thu, Feb 6, 2020 at 10:57 AM Mickael Maison <mi...@gmail.com>
wrote:

> I have not seen any new feedback nor votes.
> Bumping this thread again
>
> On Mon, Jan 27, 2020 at 3:55 PM Mickael Maison <mi...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > We are now at 4 non-binding votes but still no binding votes.
> > I have not seen any outstanding questions in the DISCUSS thread. If
> > you have any feedback, please let me know.
> >
> > Thanks
> >
> >
> > On Thu, Jan 16, 2020 at 2:03 PM M. Manna <ma...@gmail.com> wrote:
> > >
> > > MIckael,
> > >
> > >
> > >
> > > On Thu, 16 Jan 2020 at 14:01, Mickael Maison <mickael.maison@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Manna,
> > > >
> > > > In your example, the topic 'dummy' is not under replicated. It just
> > > > has 1 replica. A topic under replicated is a topic with less ISRs
> than
> > > > replicas.
> > > >
> > > > Having under replicated topics is relatively common in a Kafka
> > > > cluster, it happens everytime is broker is down. However Kafka does
> > > > not permit it to happen at topic creation. Currently at creation,
> > > > Kafka requires to have at least as many brokers as the replication
> > > > factor. This KIP addresses this limitation.
> > > >
> > > > Regarding your 2nd point. When rack awareness is enabled, Kafka tries
> > > > to distribute partitions across racks. When all brokers in a rack are
> > > > down (ie: a zone is offline), you can end up with partitions not well
> > > > distributed even with rack awareness. There are currently no easy way
> > > > to track such partitions so I decided to not attempt addressing this
> > > > issue in this KIP.
> > > >
> > > > I hope that answers your questions.
> > > >
> > >
> > >  It does and I appreciate you taking time and explaining this.
> > >
> > >  +1 (binding) if I haven't already.
> > >
> > > >
> > > >
> > > >
> > > > On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
> > > > <ka...@gmail.com> wrote:
> > > > >
> > > > > +1 (non-binding). Thanks for the KIP!
> > > > >
> > > > > On Mon, Jan 13, 2020 at 1:58 PM M. Manna <ma...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Mikael,
> > > > > >
> > > > > > Apologies for last minute question, as I just caught up with it.
> > > > Thanks for
> > > > > > your work on the KIP.
> > > > > >
> > > > > > Just trying to get your thoughts on one thing (I might have
> > > > misunderstood
> > > > > > it) - currently it's possible (even though I am strongly against
> it) to
> > > > > > create Kafka topics which are under-replicated; despite all
> brokers
> > > > being
> > > > > > online. This the the output of an intentionally under-replicated
> topic
> > > > > > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> > > > > >
> > > > > >
> > > > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$
> ./kafka-topics.sh
> > > > > > --create --topic dummy --partitions 6 --replication-factor 1
> > > > > > --bootstrap-server localhost:9092
> > > > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$
> ./kafka-topics.sh
> > > > > > --describe --topic dummy  --bootstrap-server localhost:9092
> > > > > > Topic:dummy     PartitionCount:6        ReplicationFactor:1
> > > > > >
> > > > > >
> > > >
> Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> > > > > >         Topic: dummy    Partition: 0    Leader: 3
>  Replicas: 3
> > > > > > Isr: 3
> > > > > >         Topic: dummy    Partition: 1    Leader: 1
>  Replicas: 1
> > > > > > Isr: 1
> > > > > >         Topic: dummy    Partition: 2    Leader: 2
>  Replicas: 2
> > > > > > Isr: 2
> > > > > >         Topic: dummy    Partition: 3    Leader: 3
>  Replicas: 3
> > > > > > Isr: 3
> > > > > >         Topic: dummy    Partition: 4    Leader: 1
>  Replicas: 1
> > > > > > Isr: 1
> > > > > >         Topic: dummy    Partition: 5    Leader: 2
>  Replicas: 2
> > > > > > Isr: 2
> > > > > >
> > > > > >  This is with respect to the following statement on your KIP
> (i.e.
> > > > > > under-replicated topic creation is also permitted when none is
> > > > offline):
> > > > > >
> > > > > > *but note that this may already happen (without this KIP) when
> > > > > > > topics/partitions are created while all brokers in a rack are
> offline
> > > > > > (ie:
> > > > > > > an availability zone is offline). Tracking topics/partitions
> not
> > > > > > optimally
> > > > > > > spread across all racks can be tackled in a follow up KIP.  *
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Did you mean to say that such under-replicated topics (including
> > > > > > human-created ones) will be handled in a separete KIP ?
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > >
> > > > > > On Mon, 13 Jan 2020 at 10:15, Mickael Maison <
> mickael.maison@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all.
> > > > > > >
> > > > > > > With 2.5.0 approaching, bumping this thread once more as
> feedback or
> > > > > > > votes would be nice.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <
> tbentley@redhat.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > +1 non-binding. Thanks!
> > > > > > > >
> > > > > > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > thanks for your response! That all makes perfect sense and
> I
> > > > cannot
> > > > > > > > > give any actual use cases for where what I asked about
> would be
> > > > > > useful
> > > > > > > > > :)
> > > > > > > > > It was more the idle thought if this might be low hanging
> fruit
> > > > while
> > > > > > > > > changing this anyway to avoid having to circle back later
> on and
> > > > > > > > > wanted to at least mention it.
> > > > > > > > >
> > > > > > > > > I am totally happy either way!
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Sönke
> > > > > > > > >
> > > > > > > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> > > > > > mickael.maison@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Thanks Sönke for the feedback.
> > > > > > > > > >
> > > > > > > > > > I debated this point quite a bit before deciding to base
> > > > creation
> > > > > > > > > > around "min.insync.replicas".
> > > > > > > > > >
> > > > > > > > > > For me, the goal of this KIP is to enable administrators
> to
> > > > provide
> > > > > > > > > > higher availability. In a 3 node cluster configured for
> high
> > > > > > > > > > availability (3 replicas, 2 min ISR), by enabling this
> feature,
> > > > > > > > > > clusters should be fully usable even when 1 broker is
> down.
> > > > This
> > > > > > > > > > should cover all "normal" maintenance operations like a
> rolling
> > > > > > > > > > restart or just the recovery of a broker.
> > > > > > > > > >
> > > > > > > > > > At the moment, when creating a topic/partition, the
> assumption
> > > > is
> > > > > > > that
> > > > > > > > > > the resource will be fully functioning. This KIP does not
> > > > change
> > > > > > this
> > > > > > > > > > assumption. If this is something someone wants, I think
> it
> > > > should
> > > > > > be
> > > > > > > > > > handled in a different KIP that targets that use case. By
> > > > relying
> > > > > > on
> > > > > > > > > > "min.insync.replicas", we don't break any assumptions
> the user
> > > > has
> > > > > > > and
> > > > > > > > > > this should be fully transparent from the user point of
> view.
> > > > > > > > > >
> > > > > > > > > > About "min.insync.replicas", one caveat that is not
> explicit
> > > > in the
> > > > > > > > > > KIP is that it's currently possible to create topics
> with less
> > > > > > > > > > replicas than this settings. For that reason, I think the
> > > > > > > > > > implementation will actually rely on min(replicas,
> min-isr)
> > > > instead
> > > > > > > of
> > > > > > > > > > simply min.insync.replicas. I have updated the KIP to
> > > > explicitly
> > > > > > > > > > mention this point.
> > > > > > > > > >
> > > > > > > > > > I hope that answers your question, let me know.
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Michael,
> > > > > > > > > > >
> > > > > > > > > > > that sounds like a useful addition! I can't help but
> wonder
> > > > > > > whether by
> > > > > > > > > > > leaving in the restriction that "min.insync.replicas"
> has to
> > > > be
> > > > > > > > > > > satisfied we'll be back here in a years time because
> someone
> > > > has
> > > > > > a
> > > > > > > > > > > scenario where he or she wants to go below that :)
> > > > > > > > > > > I don't have a strong opinion either way to be honest,
> just a
> > > > > > > random
> > > > > > > > > > > thought when reading the KIP.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Sönke
> > > > > > > > > > >
> > > > > > > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <
> > > > > > ryannedolan@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > +1 non-binding, thx
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Bumping this thread, I've not seen any votes or
> feedback.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > > > > > > > > <mi...@gmail.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'd like to start a vote on KIP-409: Allow
> creating
> > > > > > > > > under-replicated
> > > > > > > > > > > > > > topics and partitions
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Sönke Liebau
> > > > > > > > > > > Partner
> > > > > > > > > > > Tel. +49 179 7940878
> > > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880
> Wedel -
> > > > > > > Germany
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Sönke Liebau
> > > > > > > > > Partner
> > > > > > > > > Tel. +49 179 7940878
> > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880
> Wedel -
> > > > Germany
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
>

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Mickael Maison <mi...@gmail.com>.

I have not seen any new feedback nor votes.
Bumping this thread again

On Mon, Jan 27, 2020 at 3:55 PM Mickael Maison <mi...@gmail.com> wrote:
>
> Hi,
>
> We are now at 4 non-binding votes but still no binding votes.
> I have not seen any outstanding questions in the DISCUSS thread. If
> you have any feedback, please let me know.
>
> Thanks
>
>
> On Thu, Jan 16, 2020 at 2:03 PM M. Manna <ma...@gmail.com> wrote:
> >
> > MIckael,
> >
> >
> >
> > On Thu, 16 Jan 2020 at 14:01, Mickael Maison <mi...@gmail.com>
> > wrote:
> >
> > > Hi Manna,
> > >
> > > In your example, the topic 'dummy' is not under replicated. It just
> > > has 1 replica. A topic under replicated is a topic with less ISRs than
> > > replicas.
> > >
> > > Having under replicated topics is relatively common in a Kafka
> > > cluster, it happens everytime is broker is down. However Kafka does
> > > not permit it to happen at topic creation. Currently at creation,
> > > Kafka requires to have at least as many brokers as the replication
> > > factor. This KIP addresses this limitation.
> > >
> > > Regarding your 2nd point. When rack awareness is enabled, Kafka tries
> > > to distribute partitions across racks. When all brokers in a rack are
> > > down (ie: a zone is offline), you can end up with partitions not well
> > > distributed even with rack awareness. There are currently no easy way
> > > to track such partitions so I decided to not attempt addressing this
> > > issue in this KIP.
> > >
> > > I hope that answers your questions.
> > >
> >
> >  It does and I appreciate you taking time and explaining this.
> >
> >  +1 (binding) if I haven't already.
> >
> > >
> > >
> > >
> > > On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
> > > <ka...@gmail.com> wrote:
> > > >
> > > > +1 (non-binding). Thanks for the KIP!
> > > >
> > > > On Mon, Jan 13, 2020 at 1:58 PM M. Manna <ma...@gmail.com> wrote:
> > > >
> > > > > Hi Mikael,
> > > > >
> > > > > Apologies for last minute question, as I just caught up with it.
> > > Thanks for
> > > > > your work on the KIP.
> > > > >
> > > > > Just trying to get your thoughts on one thing (I might have
> > > misunderstood
> > > > > it) - currently it's possible (even though I am strongly against it) to
> > > > > create Kafka topics which are under-replicated; despite all brokers
> > > being
> > > > > online. This the the output of an intentionally under-replicated topic
> > > > > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> > > > >
> > > > >
> > > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > > > --create --topic dummy --partitions 6 --replication-factor 1
> > > > > --bootstrap-server localhost:9092
> > > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > > > --describe --topic dummy  --bootstrap-server localhost:9092
> > > > > Topic:dummy     PartitionCount:6        ReplicationFactor:1
> > > > >
> > > > >
> > > Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> > > > >         Topic: dummy    Partition: 0    Leader: 3       Replicas: 3
> > > > > Isr: 3
> > > > >         Topic: dummy    Partition: 1    Leader: 1       Replicas: 1
> > > > > Isr: 1
> > > > >         Topic: dummy    Partition: 2    Leader: 2       Replicas: 2
> > > > > Isr: 2
> > > > >         Topic: dummy    Partition: 3    Leader: 3       Replicas: 3
> > > > > Isr: 3
> > > > >         Topic: dummy    Partition: 4    Leader: 1       Replicas: 1
> > > > > Isr: 1
> > > > >         Topic: dummy    Partition: 5    Leader: 2       Replicas: 2
> > > > > Isr: 2
> > > > >
> > > > >  This is with respect to the following statement on your KIP (i.e.
> > > > > under-replicated topic creation is also permitted when none is
> > > offline):
> > > > >
> > > > > *but note that this may already happen (without this KIP) when
> > > > > > topics/partitions are created while all brokers in a rack are offline
> > > > > (ie:
> > > > > > an availability zone is offline). Tracking topics/partitions not
> > > > > optimally
> > > > > > spread across all racks can be tackled in a follow up KIP.  *
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Did you mean to say that such under-replicated topics (including
> > > > > human-created ones) will be handled in a separete KIP ?
> > > > >
> > > > > Regards,
> > > > >
> > > > >
> > > > > On Mon, 13 Jan 2020 at 10:15, Mickael Maison <mickael.maison@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all.
> > > > > >
> > > > > > With 2.5.0 approaching, bumping this thread once more as feedback or
> > > > > > votes would be nice.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <tb...@redhat.com>
> > > wrote:
> > > > > > >
> > > > > > > +1 non-binding. Thanks!
> > > > > > >
> > > > > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > thanks for your response! That all makes perfect sense and I
> > > cannot
> > > > > > > > give any actual use cases for where what I asked about would be
> > > > > useful
> > > > > > > > :)
> > > > > > > > It was more the idle thought if this might be low hanging fruit
> > > while
> > > > > > > > changing this anyway to avoid having to circle back later on and
> > > > > > > > wanted to at least mention it.
> > > > > > > >
> > > > > > > > I am totally happy either way!
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Sönke
> > > > > > > >
> > > > > > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> > > > > mickael.maison@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Thanks Sönke for the feedback.
> > > > > > > > >
> > > > > > > > > I debated this point quite a bit before deciding to base
> > > creation
> > > > > > > > > around "min.insync.replicas".
> > > > > > > > >
> > > > > > > > > For me, the goal of this KIP is to enable administrators to
> > > provide
> > > > > > > > > higher availability. In a 3 node cluster configured for high
> > > > > > > > > availability (3 replicas, 2 min ISR), by enabling this feature,
> > > > > > > > > clusters should be fully usable even when 1 broker is down.
> > > This
> > > > > > > > > should cover all "normal" maintenance operations like a rolling
> > > > > > > > > restart or just the recovery of a broker.
> > > > > > > > >
> > > > > > > > > At the moment, when creating a topic/partition, the assumption
> > > is
> > > > > > that
> > > > > > > > > the resource will be fully functioning. This KIP does not
> > > change
> > > > > this
> > > > > > > > > assumption. If this is something someone wants, I think it
> > > should
> > > > > be
> > > > > > > > > handled in a different KIP that targets that use case. By
> > > relying
> > > > > on
> > > > > > > > > "min.insync.replicas", we don't break any assumptions the user
> > > has
> > > > > > and
> > > > > > > > > this should be fully transparent from the user point of view.
> > > > > > > > >
> > > > > > > > > About "min.insync.replicas", one caveat that is not explicit
> > > in the
> > > > > > > > > KIP is that it's currently possible to create topics with less
> > > > > > > > > replicas than this settings. For that reason, I think the
> > > > > > > > > implementation will actually rely on min(replicas, min-isr)
> > > instead
> > > > > > of
> > > > > > > > > simply min.insync.replicas. I have updated the KIP to
> > > explicitly
> > > > > > > > > mention this point.
> > > > > > > > >
> > > > > > > > > I hope that answers your question, let me know.
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Michael,
> > > > > > > > > >
> > > > > > > > > > that sounds like a useful addition! I can't help but wonder
> > > > > > whether by
> > > > > > > > > > leaving in the restriction that "min.insync.replicas" has to
> > > be
> > > > > > > > > > satisfied we'll be back here in a years time because someone
> > > has
> > > > > a
> > > > > > > > > > scenario where he or she wants to go below that :)
> > > > > > > > > > I don't have a strong opinion either way to be honest, just a
> > > > > > random
> > > > > > > > > > thought when reading the KIP.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Sönke
> > > > > > > > > >
> > > > > > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <
> > > > > ryannedolan@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > +1 non-binding, thx
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Bumping this thread, I've not seen any votes or feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > > > > > > > <mi...@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'd like to start a vote on KIP-409: Allow creating
> > > > > > > > under-replicated
> > > > > > > > > > > > > topics and partitions
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Sönke Liebau
> > > > > > > > > > Partner
> > > > > > > > > > Tel. +49 179 7940878
> > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> > > > > > Germany
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Sönke Liebau
> > > > > > > > Partner
> > > > > > > > Tel. +49 179 7940878
> > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> > > Germany
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Mickael Maison <mi...@gmail.com>.

Hi,

We are now at 4 non-binding votes but still no binding votes.
I have not seen any outstanding questions in the DISCUSS thread. If
you have any feedback, please let me know.

Thanks


On Thu, Jan 16, 2020 at 2:03 PM M. Manna <ma...@gmail.com> wrote:
>
> MIckael,
>
>
>
> On Thu, 16 Jan 2020 at 14:01, Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi Manna,
> >
> > In your example, the topic 'dummy' is not under replicated. It just
> > has 1 replica. A topic under replicated is a topic with less ISRs than
> > replicas.
> >
> > Having under replicated topics is relatively common in a Kafka
> > cluster, it happens everytime is broker is down. However Kafka does
> > not permit it to happen at topic creation. Currently at creation,
> > Kafka requires to have at least as many brokers as the replication
> > factor. This KIP addresses this limitation.
> >
> > Regarding your 2nd point. When rack awareness is enabled, Kafka tries
> > to distribute partitions across racks. When all brokers in a rack are
> > down (ie: a zone is offline), you can end up with partitions not well
> > distributed even with rack awareness. There are currently no easy way
> > to track such partitions so I decided to not attempt addressing this
> > issue in this KIP.
> >
> > I hope that answers your questions.
> >
>
>  It does and I appreciate you taking time and explaining this.
>
>  +1 (binding) if I haven't already.
>
> >
> >
> >
> > On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
> > <ka...@gmail.com> wrote:
> > >
> > > +1 (non-binding). Thanks for the KIP!
> > >
> > > On Mon, Jan 13, 2020 at 1:58 PM M. Manna <ma...@gmail.com> wrote:
> > >
> > > > Hi Mikael,
> > > >
> > > > Apologies for last minute question, as I just caught up with it.
> > Thanks for
> > > > your work on the KIP.
> > > >
> > > > Just trying to get your thoughts on one thing (I might have
> > misunderstood
> > > > it) - currently it's possible (even though I am strongly against it) to
> > > > create Kafka topics which are under-replicated; despite all brokers
> > being
> > > > online. This the the output of an intentionally under-replicated topic
> > > > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> > > >
> > > >
> > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > > --create --topic dummy --partitions 6 --replication-factor 1
> > > > --bootstrap-server localhost:9092
> > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > > --describe --topic dummy  --bootstrap-server localhost:9092
> > > > Topic:dummy     PartitionCount:6        ReplicationFactor:1
> > > >
> > > >
> > Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> > > >         Topic: dummy    Partition: 0    Leader: 3       Replicas: 3
> > > > Isr: 3
> > > >         Topic: dummy    Partition: 1    Leader: 1       Replicas: 1
> > > > Isr: 1
> > > >         Topic: dummy    Partition: 2    Leader: 2       Replicas: 2
> > > > Isr: 2
> > > >         Topic: dummy    Partition: 3    Leader: 3       Replicas: 3
> > > > Isr: 3
> > > >         Topic: dummy    Partition: 4    Leader: 1       Replicas: 1
> > > > Isr: 1
> > > >         Topic: dummy    Partition: 5    Leader: 2       Replicas: 2
> > > > Isr: 2
> > > >
> > > >  This is with respect to the following statement on your KIP (i.e.
> > > > under-replicated topic creation is also permitted when none is
> > offline):
> > > >
> > > > *but note that this may already happen (without this KIP) when
> > > > > topics/partitions are created while all brokers in a rack are offline
> > > > (ie:
> > > > > an availability zone is offline). Tracking topics/partitions not
> > > > optimally
> > > > > spread across all racks can be tackled in a follow up KIP.  *
> > > >
> > > >
> > > >
> > > >
> > > > Did you mean to say that such under-replicated topics (including
> > > > human-created ones) will be handled in a separete KIP ?
> > > >
> > > > Regards,
> > > >
> > > >
> > > > On Mon, 13 Jan 2020 at 10:15, Mickael Maison <mickael.maison@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi all.
> > > > >
> > > > > With 2.5.0 approaching, bumping this thread once more as feedback or
> > > > > votes would be nice.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <tb...@redhat.com>
> > wrote:
> > > > > >
> > > > > > +1 non-binding. Thanks!
> > > > > >
> > > > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > > > <so...@opencore.com.invalid> wrote:
> > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > thanks for your response! That all makes perfect sense and I
> > cannot
> > > > > > > give any actual use cases for where what I asked about would be
> > > > useful
> > > > > > > :)
> > > > > > > It was more the idle thought if this might be low hanging fruit
> > while
> > > > > > > changing this anyway to avoid having to circle back later on and
> > > > > > > wanted to at least mention it.
> > > > > > >
> > > > > > > I am totally happy either way!
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Sönke
> > > > > > >
> > > > > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> > > > mickael.maison@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Thanks Sönke for the feedback.
> > > > > > > >
> > > > > > > > I debated this point quite a bit before deciding to base
> > creation
> > > > > > > > around "min.insync.replicas".
> > > > > > > >
> > > > > > > > For me, the goal of this KIP is to enable administrators to
> > provide
> > > > > > > > higher availability. In a 3 node cluster configured for high
> > > > > > > > availability (3 replicas, 2 min ISR), by enabling this feature,
> > > > > > > > clusters should be fully usable even when 1 broker is down.
> > This
> > > > > > > > should cover all "normal" maintenance operations like a rolling
> > > > > > > > restart or just the recovery of a broker.
> > > > > > > >
> > > > > > > > At the moment, when creating a topic/partition, the assumption
> > is
> > > > > that
> > > > > > > > the resource will be fully functioning. This KIP does not
> > change
> > > > this
> > > > > > > > assumption. If this is something someone wants, I think it
> > should
> > > > be
> > > > > > > > handled in a different KIP that targets that use case. By
> > relying
> > > > on
> > > > > > > > "min.insync.replicas", we don't break any assumptions the user
> > has
> > > > > and
> > > > > > > > this should be fully transparent from the user point of view.
> > > > > > > >
> > > > > > > > About "min.insync.replicas", one caveat that is not explicit
> > in the
> > > > > > > > KIP is that it's currently possible to create topics with less
> > > > > > > > replicas than this settings. For that reason, I think the
> > > > > > > > implementation will actually rely on min(replicas, min-isr)
> > instead
> > > > > of
> > > > > > > > simply min.insync.replicas. I have updated the KIP to
> > explicitly
> > > > > > > > mention this point.
> > > > > > > >
> > > > > > > > I hope that answers your question, let me know.
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > > > >
> > > > > > > > > Hi Michael,
> > > > > > > > >
> > > > > > > > > that sounds like a useful addition! I can't help but wonder
> > > > > whether by
> > > > > > > > > leaving in the restriction that "min.insync.replicas" has to
> > be
> > > > > > > > > satisfied we'll be back here in a years time because someone
> > has
> > > > a
> > > > > > > > > scenario where he or she wants to go below that :)
> > > > > > > > > I don't have a strong opinion either way to be honest, just a
> > > > > random
> > > > > > > > > thought when reading the KIP.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Sönke
> > > > > > > > >
> > > > > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <
> > > > ryannedolan@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1 non-binding, thx
> > > > > > > > > >
> > > > > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Bumping this thread, I've not seen any votes or feedback.
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > > > > > > <mi...@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'd like to start a vote on KIP-409: Allow creating
> > > > > > > under-replicated
> > > > > > > > > > > > topics and partitions
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Sönke Liebau
> > > > > > > > > Partner
> > > > > > > > > Tel. +49 179 7940878
> > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> > > > > Germany
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Sönke Liebau
> > > > > > > Partner
> > > > > > > Tel. +49 179 7940878
> > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> > Germany
> > > > > > >
> > > > > > >
> > > > >
> > > >
> >

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by "M. Manna" <ma...@gmail.com>.

MIckael,



On Thu, 16 Jan 2020 at 14:01, Mickael Maison <mi...@gmail.com>
wrote:

> Hi Manna,
>
> In your example, the topic 'dummy' is not under replicated. It just
> has 1 replica. A topic under replicated is a topic with less ISRs than
> replicas.
>
> Having under replicated topics is relatively common in a Kafka
> cluster, it happens everytime is broker is down. However Kafka does
> not permit it to happen at topic creation. Currently at creation,
> Kafka requires to have at least as many brokers as the replication
> factor. This KIP addresses this limitation.
>
> Regarding your 2nd point. When rack awareness is enabled, Kafka tries
> to distribute partitions across racks. When all brokers in a rack are
> down (ie: a zone is offline), you can end up with partitions not well
> distributed even with rack awareness. There are currently no easy way
> to track such partitions so I decided to not attempt addressing this
> issue in this KIP.
>
> I hope that answers your questions.
>

 It does and I appreciate you taking time and explaining this.

 +1 (binding) if I haven't already.

>
>
>
> On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
> <ka...@gmail.com> wrote:
> >
> > +1 (non-binding). Thanks for the KIP!
> >
> > On Mon, Jan 13, 2020 at 1:58 PM M. Manna <ma...@gmail.com> wrote:
> >
> > > Hi Mikael,
> > >
> > > Apologies for last minute question, as I just caught up with it.
> Thanks for
> > > your work on the KIP.
> > >
> > > Just trying to get your thoughts on one thing (I might have
> misunderstood
> > > it) - currently it's possible (even though I am strongly against it) to
> > > create Kafka topics which are under-replicated; despite all brokers
> being
> > > online. This the the output of an intentionally under-replicated topic
> > > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> > >
> > >
> > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > --create --topic dummy --partitions 6 --replication-factor 1
> > > --bootstrap-server localhost:9092
> > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > --describe --topic dummy  --bootstrap-server localhost:9092
> > > Topic:dummy     PartitionCount:6        ReplicationFactor:1
> > >
> > >
> Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> > >         Topic: dummy    Partition: 0    Leader: 3       Replicas: 3
> > > Isr: 3
> > >         Topic: dummy    Partition: 1    Leader: 1       Replicas: 1
> > > Isr: 1
> > >         Topic: dummy    Partition: 2    Leader: 2       Replicas: 2
> > > Isr: 2
> > >         Topic: dummy    Partition: 3    Leader: 3       Replicas: 3
> > > Isr: 3
> > >         Topic: dummy    Partition: 4    Leader: 1       Replicas: 1
> > > Isr: 1
> > >         Topic: dummy    Partition: 5    Leader: 2       Replicas: 2
> > > Isr: 2
> > >
> > >  This is with respect to the following statement on your KIP (i.e.
> > > under-replicated topic creation is also permitted when none is
> offline):
> > >
> > > *but note that this may already happen (without this KIP) when
> > > > topics/partitions are created while all brokers in a rack are offline
> > > (ie:
> > > > an availability zone is offline). Tracking topics/partitions not
> > > optimally
> > > > spread across all racks can be tackled in a follow up KIP.  *
> > >
> > >
> > >
> > >
> > > Did you mean to say that such under-replicated topics (including
> > > human-created ones) will be handled in a separete KIP ?
> > >
> > > Regards,
> > >
> > >
> > > On Mon, 13 Jan 2020 at 10:15, Mickael Maison <mickael.maison@gmail.com
> >
> > > wrote:
> > >
> > > > Hi all.
> > > >
> > > > With 2.5.0 approaching, bumping this thread once more as feedback or
> > > > votes would be nice.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <tb...@redhat.com>
> wrote:
> > > > >
> > > > > +1 non-binding. Thanks!
> > > > >
> > > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > > <so...@opencore.com.invalid> wrote:
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > thanks for your response! That all makes perfect sense and I
> cannot
> > > > > > give any actual use cases for where what I asked about would be
> > > useful
> > > > > > :)
> > > > > > It was more the idle thought if this might be low hanging fruit
> while
> > > > > > changing this anyway to avoid having to circle back later on and
> > > > > > wanted to at least mention it.
> > > > > >
> > > > > > I am totally happy either way!
> > > > > >
> > > > > > Best regards,
> > > > > > Sönke
> > > > > >
> > > > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> > > mickael.maison@gmail.com
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Thanks Sönke for the feedback.
> > > > > > >
> > > > > > > I debated this point quite a bit before deciding to base
> creation
> > > > > > > around "min.insync.replicas".
> > > > > > >
> > > > > > > For me, the goal of this KIP is to enable administrators to
> provide
> > > > > > > higher availability. In a 3 node cluster configured for high
> > > > > > > availability (3 replicas, 2 min ISR), by enabling this feature,
> > > > > > > clusters should be fully usable even when 1 broker is down.
> This
> > > > > > > should cover all "normal" maintenance operations like a rolling
> > > > > > > restart or just the recovery of a broker.
> > > > > > >
> > > > > > > At the moment, when creating a topic/partition, the assumption
> is
> > > > that
> > > > > > > the resource will be fully functioning. This KIP does not
> change
> > > this
> > > > > > > assumption. If this is something someone wants, I think it
> should
> > > be
> > > > > > > handled in a different KIP that targets that use case. By
> relying
> > > on
> > > > > > > "min.insync.replicas", we don't break any assumptions the user
> has
> > > > and
> > > > > > > this should be fully transparent from the user point of view.
> > > > > > >
> > > > > > > About "min.insync.replicas", one caveat that is not explicit
> in the
> > > > > > > KIP is that it's currently possible to create topics with less
> > > > > > > replicas than this settings. For that reason, I think the
> > > > > > > implementation will actually rely on min(replicas, min-isr)
> instead
> > > > of
> > > > > > > simply min.insync.replicas. I have updated the KIP to
> explicitly
> > > > > > > mention this point.
> > > > > > >
> > > > > > > I hope that answers your question, let me know.
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > > >
> > > > > > > > Hi Michael,
> > > > > > > >
> > > > > > > > that sounds like a useful addition! I can't help but wonder
> > > > whether by
> > > > > > > > leaving in the restriction that "min.insync.replicas" has to
> be
> > > > > > > > satisfied we'll be back here in a years time because someone
> has
> > > a
> > > > > > > > scenario where he or she wants to go below that :)
> > > > > > > > I don't have a strong opinion either way to be honest, just a
> > > > random
> > > > > > > > thought when reading the KIP.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Sönke
> > > > > > > >
> > > > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <
> > > ryannedolan@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > +1 non-binding, thx
> > > > > > > > >
> > > > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > > > > > mickael.maison@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Bumping this thread, I've not seen any votes or feedback.
> > > > > > > > > >
> > > > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > > > > > <mi...@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I'd like to start a vote on KIP-409: Allow creating
> > > > > > under-replicated
> > > > > > > > > > > topics and partitions
> > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Sönke Liebau
> > > > > > > > Partner
> > > > > > > > Tel. +49 179 7940878
> > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> > > > Germany
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sönke Liebau
> > > > > > Partner
> > > > > > Tel. +49 179 7940878
> > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> Germany
> > > > > >
> > > > > >
> > > >
> > >
>

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Mickael Maison <mi...@gmail.com>.

Hi Manna,

In your example, the topic 'dummy' is not under replicated. It just
has 1 replica. A topic under replicated is a topic with less ISRs than
replicas.

Having under replicated topics is relatively common in a Kafka
cluster, it happens everytime is broker is down. However Kafka does
not permit it to happen at topic creation. Currently at creation,
Kafka requires to have at least as many brokers as the replication
factor. This KIP addresses this limitation.

Regarding your 2nd point. When rack awareness is enabled, Kafka tries
to distribute partitions across racks. When all brokers in a rack are
down (ie: a zone is offline), you can end up with partitions not well
distributed even with rack awareness. There are currently no easy way
to track such partitions so I decided to not attempt addressing this
issue in this KIP.

I hope that answers your questions.



On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
<ka...@gmail.com> wrote:
>
> +1 (non-binding). Thanks for the KIP!
>
> On Mon, Jan 13, 2020 at 1:58 PM M. Manna <ma...@gmail.com> wrote:
>
> > Hi Mikael,
> >
> > Apologies for last minute question, as I just caught up with it. Thanks for
> > your work on the KIP.
> >
> > Just trying to get your thoughts on one thing (I might have misunderstood
> > it) - currently it's possible (even though I am strongly against it) to
> > create Kafka topics which are under-replicated; despite all brokers being
> > online. This the the output of an intentionally under-replicated topic
> > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> >
> >
> > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > --create --topic dummy --partitions 6 --replication-factor 1
> > --bootstrap-server localhost:9092
> > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > --describe --topic dummy  --bootstrap-server localhost:9092
> > Topic:dummy     PartitionCount:6        ReplicationFactor:1
> >
> > Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> >         Topic: dummy    Partition: 0    Leader: 3       Replicas: 3
> > Isr: 3
> >         Topic: dummy    Partition: 1    Leader: 1       Replicas: 1
> > Isr: 1
> >         Topic: dummy    Partition: 2    Leader: 2       Replicas: 2
> > Isr: 2
> >         Topic: dummy    Partition: 3    Leader: 3       Replicas: 3
> > Isr: 3
> >         Topic: dummy    Partition: 4    Leader: 1       Replicas: 1
> > Isr: 1
> >         Topic: dummy    Partition: 5    Leader: 2       Replicas: 2
> > Isr: 2
> >
> >  This is with respect to the following statement on your KIP (i.e.
> > under-replicated topic creation is also permitted when none is offline):
> >
> > *but note that this may already happen (without this KIP) when
> > > topics/partitions are created while all brokers in a rack are offline
> > (ie:
> > > an availability zone is offline). Tracking topics/partitions not
> > optimally
> > > spread across all racks can be tackled in a follow up KIP.  *
> >
> >
> >
> >
> > Did you mean to say that such under-replicated topics (including
> > human-created ones) will be handled in a separete KIP ?
> >
> > Regards,
> >
> >
> > On Mon, 13 Jan 2020 at 10:15, Mickael Maison <mi...@gmail.com>
> > wrote:
> >
> > > Hi all.
> > >
> > > With 2.5.0 approaching, bumping this thread once more as feedback or
> > > votes would be nice.
> > >
> > > Thanks
> > >
> > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <tb...@redhat.com> wrote:
> > > >
> > > > +1 non-binding. Thanks!
> > > >
> > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > <so...@opencore.com.invalid> wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > thanks for your response! That all makes perfect sense and I cannot
> > > > > give any actual use cases for where what I asked about would be
> > useful
> > > > > :)
> > > > > It was more the idle thought if this might be low hanging fruit while
> > > > > changing this anyway to avoid having to circle back later on and
> > > > > wanted to at least mention it.
> > > > >
> > > > > I am totally happy either way!
> > > > >
> > > > > Best regards,
> > > > > Sönke
> > > > >
> > > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> > mickael.maison@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Thanks Sönke for the feedback.
> > > > > >
> > > > > > I debated this point quite a bit before deciding to base creation
> > > > > > around "min.insync.replicas".
> > > > > >
> > > > > > For me, the goal of this KIP is to enable administrators to provide
> > > > > > higher availability. In a 3 node cluster configured for high
> > > > > > availability (3 replicas, 2 min ISR), by enabling this feature,
> > > > > > clusters should be fully usable even when 1 broker is down. This
> > > > > > should cover all "normal" maintenance operations like a rolling
> > > > > > restart or just the recovery of a broker.
> > > > > >
> > > > > > At the moment, when creating a topic/partition, the assumption is
> > > that
> > > > > > the resource will be fully functioning. This KIP does not change
> > this
> > > > > > assumption. If this is something someone wants, I think it should
> > be
> > > > > > handled in a different KIP that targets that use case. By relying
> > on
> > > > > > "min.insync.replicas", we don't break any assumptions the user has
> > > and
> > > > > > this should be fully transparent from the user point of view.
> > > > > >
> > > > > > About "min.insync.replicas", one caveat that is not explicit in the
> > > > > > KIP is that it's currently possible to create topics with less
> > > > > > replicas than this settings. For that reason, I think the
> > > > > > implementation will actually rely on min(replicas, min-isr) instead
> > > of
> > > > > > simply min.insync.replicas. I have updated the KIP to explicitly
> > > > > > mention this point.
> > > > > >
> > > > > > I hope that answers your question, let me know.
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > >
> > > > > > > Hi Michael,
> > > > > > >
> > > > > > > that sounds like a useful addition! I can't help but wonder
> > > whether by
> > > > > > > leaving in the restriction that "min.insync.replicas" has to be
> > > > > > > satisfied we'll be back here in a years time because someone has
> > a
> > > > > > > scenario where he or she wants to go below that :)
> > > > > > > I don't have a strong opinion either way to be honest, just a
> > > random
> > > > > > > thought when reading the KIP.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Sönke
> > > > > > >
> > > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <
> > ryannedolan@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > +1 non-binding, thx
> > > > > > > >
> > > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > > > > mickael.maison@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Bumping this thread, I've not seen any votes or feedback.
> > > > > > > > >
> > > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > > > > <mi...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'd like to start a vote on KIP-409: Allow creating
> > > > > under-replicated
> > > > > > > > > > topics and partitions
> > > > > > > > > >
> > > > > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Sönke Liebau
> > > > > > > Partner
> > > > > > > Tel. +49 179 7940878
> > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> > > Germany
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sönke Liebau
> > > > > Partner
> > > > > Tel. +49 179 7940878
> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > > > >
> > > > >
> > >
> >

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Kamal Chandraprakash <ka...@gmail.com>.

+1 (non-binding). Thanks for the KIP!

On Mon, Jan 13, 2020 at 1:58 PM M. Manna <ma...@gmail.com> wrote:

> Hi Mikael,
>
> Apologies for last minute question, as I just caught up with it. Thanks for
> your work on the KIP.
>
> Just trying to get your thoughts on one thing (I might have misunderstood
> it) - currently it's possible (even though I am strongly against it) to
> create Kafka topics which are under-replicated; despite all brokers being
> online. This the the output of an intentionally under-replicated topic
> "dummy" with p=6 and RF=1 (with a 3 node cluster)
>
>
> virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> --create --topic dummy --partitions 6 --replication-factor 1
> --bootstrap-server localhost:9092
> virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> --describe --topic dummy  --bootstrap-server localhost:9092
> Topic:dummy     PartitionCount:6        ReplicationFactor:1
>
> Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
>         Topic: dummy    Partition: 0    Leader: 3       Replicas: 3
> Isr: 3
>         Topic: dummy    Partition: 1    Leader: 1       Replicas: 1
> Isr: 1
>         Topic: dummy    Partition: 2    Leader: 2       Replicas: 2
> Isr: 2
>         Topic: dummy    Partition: 3    Leader: 3       Replicas: 3
> Isr: 3
>         Topic: dummy    Partition: 4    Leader: 1       Replicas: 1
> Isr: 1
>         Topic: dummy    Partition: 5    Leader: 2       Replicas: 2
> Isr: 2
>
>  This is with respect to the following statement on your KIP (i.e.
> under-replicated topic creation is also permitted when none is offline):
>
> *but note that this may already happen (without this KIP) when
> > topics/partitions are created while all brokers in a rack are offline
> (ie:
> > an availability zone is offline). Tracking topics/partitions not
> optimally
> > spread across all racks can be tackled in a follow up KIP.  *
>
>
>
>
> Did you mean to say that such under-replicated topics (including
> human-created ones) will be handled in a separete KIP ?
>
> Regards,
>
>
> On Mon, 13 Jan 2020 at 10:15, Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi all.
> >
> > With 2.5.0 approaching, bumping this thread once more as feedback or
> > votes would be nice.
> >
> > Thanks
> >
> > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <tb...@redhat.com> wrote:
> > >
> > > +1 non-binding. Thanks!
> > >
> > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > <so...@opencore.com.invalid> wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > thanks for your response! That all makes perfect sense and I cannot
> > > > give any actual use cases for where what I asked about would be
> useful
> > > > :)
> > > > It was more the idle thought if this might be low hanging fruit while
> > > > changing this anyway to avoid having to circle back later on and
> > > > wanted to at least mention it.
> > > >
> > > > I am totally happy either way!
> > > >
> > > > Best regards,
> > > > Sönke
> > > >
> > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> mickael.maison@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > > Thanks Sönke for the feedback.
> > > > >
> > > > > I debated this point quite a bit before deciding to base creation
> > > > > around "min.insync.replicas".
> > > > >
> > > > > For me, the goal of this KIP is to enable administrators to provide
> > > > > higher availability. In a 3 node cluster configured for high
> > > > > availability (3 replicas, 2 min ISR), by enabling this feature,
> > > > > clusters should be fully usable even when 1 broker is down. This
> > > > > should cover all "normal" maintenance operations like a rolling
> > > > > restart or just the recovery of a broker.
> > > > >
> > > > > At the moment, when creating a topic/partition, the assumption is
> > that
> > > > > the resource will be fully functioning. This KIP does not change
> this
> > > > > assumption. If this is something someone wants, I think it should
> be
> > > > > handled in a different KIP that targets that use case. By relying
> on
> > > > > "min.insync.replicas", we don't break any assumptions the user has
> > and
> > > > > this should be fully transparent from the user point of view.
> > > > >
> > > > > About "min.insync.replicas", one caveat that is not explicit in the
> > > > > KIP is that it's currently possible to create topics with less
> > > > > replicas than this settings. For that reason, I think the
> > > > > implementation will actually rely on min(replicas, min-isr) instead
> > of
> > > > > simply min.insync.replicas. I have updated the KIP to explicitly
> > > > > mention this point.
> > > > >
> > > > > I hope that answers your question, let me know.
> > > > > Thanks
> > > > >
> > > > >
> > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > > <so...@opencore.com.invalid> wrote:
> > > > > >
> > > > > > Hi Michael,
> > > > > >
> > > > > > that sounds like a useful addition! I can't help but wonder
> > whether by
> > > > > > leaving in the restriction that "min.insync.replicas" has to be
> > > > > > satisfied we'll be back here in a years time because someone has
> a
> > > > > > scenario where he or she wants to go below that :)
> > > > > > I don't have a strong opinion either way to be honest, just a
> > random
> > > > > > thought when reading the KIP.
> > > > > >
> > > > > > Best regards,
> > > > > > Sönke
> > > > > >
> > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <
> ryannedolan@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > +1 non-binding, thx
> > > > > > >
> > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > > > mickael.maison@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Bumping this thread, I've not seen any votes or feedback.
> > > > > > > >
> > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > > > <mi...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'd like to start a vote on KIP-409: Allow creating
> > > > under-replicated
> > > > > > > > > topics and partitions
> > > > > > > > >
> > > > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sönke Liebau
> > > > > > Partner
> > > > > > Tel. +49 179 7940878
> > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> > Germany
> > > >
> > > >
> > > >
> > > > --
> > > > Sönke Liebau
> > > > Partner
> > > > Tel. +49 179 7940878
> > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > > >
> > > >
> >
>

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by "M. Manna" <ma...@gmail.com>.

Hi Mikael,

Apologies for last minute question, as I just caught up with it. Thanks for
your work on the KIP.

Just trying to get your thoughts on one thing (I might have misunderstood
it) - currently it's possible (even though I am strongly against it) to
create Kafka topics which are under-replicated; despite all brokers being
online. This the the output of an intentionally under-replicated topic
"dummy" with p=6 and RF=1 (with a 3 node cluster)


virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
--create --topic dummy --partitions 6 --replication-factor 1
--bootstrap-server localhost:9092
virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
--describe --topic dummy  --bootstrap-server localhost:9092
Topic:dummy     PartitionCount:6        ReplicationFactor:1
Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
        Topic: dummy    Partition: 0    Leader: 3       Replicas: 3
Isr: 3
        Topic: dummy    Partition: 1    Leader: 1       Replicas: 1
Isr: 1
        Topic: dummy    Partition: 2    Leader: 2       Replicas: 2
Isr: 2
        Topic: dummy    Partition: 3    Leader: 3       Replicas: 3
Isr: 3
        Topic: dummy    Partition: 4    Leader: 1       Replicas: 1
Isr: 1
        Topic: dummy    Partition: 5    Leader: 2       Replicas: 2
Isr: 2

 This is with respect to the following statement on your KIP (i.e.
under-replicated topic creation is also permitted when none is offline):

*but note that this may already happen (without this KIP) when
> topics/partitions are created while all brokers in a rack are offline (ie:
> an availability zone is offline). Tracking topics/partitions not optimally
> spread across all racks can be tackled in a follow up KIP.  *




Did you mean to say that such under-replicated topics (including
human-created ones) will be handled in a separete KIP ?

Regards,


On Mon, 13 Jan 2020 at 10:15, Mickael Maison <mi...@gmail.com>
wrote:

> Hi all.
>
> With 2.5.0 approaching, bumping this thread once more as feedback or
> votes would be nice.
>
> Thanks
>
> On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <tb...@redhat.com> wrote:
> >
> > +1 non-binding. Thanks!
> >
> > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > <so...@opencore.com.invalid> wrote:
> >
> > > Hi Mickael,
> > >
> > > thanks for your response! That all makes perfect sense and I cannot
> > > give any actual use cases for where what I asked about would be useful
> > > :)
> > > It was more the idle thought if this might be low hanging fruit while
> > > changing this anyway to avoid having to circle back later on and
> > > wanted to at least mention it.
> > >
> > > I am totally happy either way!
> > >
> > > Best regards,
> > > Sönke
> > >
> > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <mickael.maison@gmail.com
> >
> > > wrote:
> > > >
> > > > Thanks Sönke for the feedback.
> > > >
> > > > I debated this point quite a bit before deciding to base creation
> > > > around "min.insync.replicas".
> > > >
> > > > For me, the goal of this KIP is to enable administrators to provide
> > > > higher availability. In a 3 node cluster configured for high
> > > > availability (3 replicas, 2 min ISR), by enabling this feature,
> > > > clusters should be fully usable even when 1 broker is down. This
> > > > should cover all "normal" maintenance operations like a rolling
> > > > restart or just the recovery of a broker.
> > > >
> > > > At the moment, when creating a topic/partition, the assumption is
> that
> > > > the resource will be fully functioning. This KIP does not change this
> > > > assumption. If this is something someone wants, I think it should be
> > > > handled in a different KIP that targets that use case. By relying on
> > > > "min.insync.replicas", we don't break any assumptions the user has
> and
> > > > this should be fully transparent from the user point of view.
> > > >
> > > > About "min.insync.replicas", one caveat that is not explicit in the
> > > > KIP is that it's currently possible to create topics with less
> > > > replicas than this settings. For that reason, I think the
> > > > implementation will actually rely on min(replicas, min-isr) instead
> of
> > > > simply min.insync.replicas. I have updated the KIP to explicitly
> > > > mention this point.
> > > >
> > > > I hope that answers your question, let me know.
> > > > Thanks
> > > >
> > > >
> > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > <so...@opencore.com.invalid> wrote:
> > > > >
> > > > > Hi Michael,
> > > > >
> > > > > that sounds like a useful addition! I can't help but wonder
> whether by
> > > > > leaving in the restriction that "min.insync.replicas" has to be
> > > > > satisfied we'll be back here in a years time because someone has a
> > > > > scenario where he or she wants to go below that :)
> > > > > I don't have a strong opinion either way to be honest, just a
> random
> > > > > thought when reading the KIP.
> > > > >
> > > > > Best regards,
> > > > > Sönke
> > > > >
> > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <ry...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > +1 non-binding, thx
> > > > > >
> > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > > mickael.maison@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Bumping this thread, I've not seen any votes or feedback.
> > > > > > >
> > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > > <mi...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'd like to start a vote on KIP-409: Allow creating
> > > under-replicated
> > > > > > > > topics and partitions
> > > > > > > >
> > > > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > >
> > > > > > > > Thanks
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sönke Liebau
> > > > > Partner
> > > > > Tel. +49 179 7940878
> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> Germany
> > >
> > >
> > >
> > > --
> > > Sönke Liebau
> > > Partner
> > > Tel. +49 179 7940878
> > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > >
> > >
>

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Mickael Maison <mi...@gmail.com>.

Hi all.

With 2.5.0 approaching, bumping this thread once more as feedback or
votes would be nice.

Thanks

On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <tb...@redhat.com> wrote:
>
> +1 non-binding. Thanks!
>
> On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> <so...@opencore.com.invalid> wrote:
>
> > Hi Mickael,
> >
> > thanks for your response! That all makes perfect sense and I cannot
> > give any actual use cases for where what I asked about would be useful
> > :)
> > It was more the idle thought if this might be low hanging fruit while
> > changing this anyway to avoid having to circle back later on and
> > wanted to at least mention it.
> >
> > I am totally happy either way!
> >
> > Best regards,
> > Sönke
> >
> > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <mi...@gmail.com>
> > wrote:
> > >
> > > Thanks Sönke for the feedback.
> > >
> > > I debated this point quite a bit before deciding to base creation
> > > around "min.insync.replicas".
> > >
> > > For me, the goal of this KIP is to enable administrators to provide
> > > higher availability. In a 3 node cluster configured for high
> > > availability (3 replicas, 2 min ISR), by enabling this feature,
> > > clusters should be fully usable even when 1 broker is down. This
> > > should cover all "normal" maintenance operations like a rolling
> > > restart or just the recovery of a broker.
> > >
> > > At the moment, when creating a topic/partition, the assumption is that
> > > the resource will be fully functioning. This KIP does not change this
> > > assumption. If this is something someone wants, I think it should be
> > > handled in a different KIP that targets that use case. By relying on
> > > "min.insync.replicas", we don't break any assumptions the user has and
> > > this should be fully transparent from the user point of view.
> > >
> > > About "min.insync.replicas", one caveat that is not explicit in the
> > > KIP is that it's currently possible to create topics with less
> > > replicas than this settings. For that reason, I think the
> > > implementation will actually rely on min(replicas, min-isr) instead of
> > > simply min.insync.replicas. I have updated the KIP to explicitly
> > > mention this point.
> > >
> > > I hope that answers your question, let me know.
> > > Thanks
> > >
> > >
> > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > <so...@opencore.com.invalid> wrote:
> > > >
> > > > Hi Michael,
> > > >
> > > > that sounds like a useful addition! I can't help but wonder whether by
> > > > leaving in the restriction that "min.insync.replicas" has to be
> > > > satisfied we'll be back here in a years time because someone has a
> > > > scenario where he or she wants to go below that :)
> > > > I don't have a strong opinion either way to be honest, just a random
> > > > thought when reading the KIP.
> > > >
> > > > Best regards,
> > > > Sönke
> > > >
> > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <ry...@gmail.com>
> > wrote:
> > > > >
> > > > > +1 non-binding, thx
> > > > >
> > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > mickael.maison@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Bumping this thread, I've not seen any votes or feedback.
> > > > > >
> > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > <mi...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to start a vote on KIP-409: Allow creating
> > under-replicated
> > > > > > > topics and partitions
> > > > > > >
> > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > >
> > > > > > > Thanks
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sönke Liebau
> > > > Partner
> > > > Tel. +49 179 7940878
> > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> >
> >
> >
> > --
> > Sönke Liebau
> > Partner
> > Tel. +49 179 7940878
> > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> >
> >

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Tom Bentley <tb...@redhat.com>.

+1 non-binding. Thanks!

On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
<so...@opencore.com.invalid> wrote:

> Hi Mickael,
>
> thanks for your response! That all makes perfect sense and I cannot
> give any actual use cases for where what I asked about would be useful
> :)
> It was more the idle thought if this might be low hanging fruit while
> changing this anyway to avoid having to circle back later on and
> wanted to at least mention it.
>
> I am totally happy either way!
>
> Best regards,
> Sönke
>
> On Wed, 18 Dec 2019 at 11:20, Mickael Maison <mi...@gmail.com>
> wrote:
> >
> > Thanks Sönke for the feedback.
> >
> > I debated this point quite a bit before deciding to base creation
> > around "min.insync.replicas".
> >
> > For me, the goal of this KIP is to enable administrators to provide
> > higher availability. In a 3 node cluster configured for high
> > availability (3 replicas, 2 min ISR), by enabling this feature,
> > clusters should be fully usable even when 1 broker is down. This
> > should cover all "normal" maintenance operations like a rolling
> > restart or just the recovery of a broker.
> >
> > At the moment, when creating a topic/partition, the assumption is that
> > the resource will be fully functioning. This KIP does not change this
> > assumption. If this is something someone wants, I think it should be
> > handled in a different KIP that targets that use case. By relying on
> > "min.insync.replicas", we don't break any assumptions the user has and
> > this should be fully transparent from the user point of view.
> >
> > About "min.insync.replicas", one caveat that is not explicit in the
> > KIP is that it's currently possible to create topics with less
> > replicas than this settings. For that reason, I think the
> > implementation will actually rely on min(replicas, min-isr) instead of
> > simply min.insync.replicas. I have updated the KIP to explicitly
> > mention this point.
> >
> > I hope that answers your question, let me know.
> > Thanks
> >
> >
> > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > <so...@opencore.com.invalid> wrote:
> > >
> > > Hi Michael,
> > >
> > > that sounds like a useful addition! I can't help but wonder whether by
> > > leaving in the restriction that "min.insync.replicas" has to be
> > > satisfied we'll be back here in a years time because someone has a
> > > scenario where he or she wants to go below that :)
> > > I don't have a strong opinion either way to be honest, just a random
> > > thought when reading the KIP.
> > >
> > > Best regards,
> > > Sönke
> > >
> > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <ry...@gmail.com>
> wrote:
> > > >
> > > > +1 non-binding, thx
> > > >
> > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> mickael.maison@gmail.com>
> > > > wrote:
> > > >
> > > > > Bumping this thread, I've not seen any votes or feedback.
> > > > >
> > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > <mi...@gmail.com> wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start a vote on KIP-409: Allow creating
> under-replicated
> > > > > > topics and partitions
> > > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > >
> > > > > > Thanks
> > > > >
> > >
> > >
> > >
> > > --
> > > Sönke Liebau
> > > Partner
> > > Tel. +49 179 7940878
> > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>
>

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Sönke Liebau <so...@opencore.com.INVALID>.

Hi Mickael,

thanks for your response! That all makes perfect sense and I cannot
give any actual use cases for where what I asked about would be useful
:)
It was more the idle thought if this might be low hanging fruit while
changing this anyway to avoid having to circle back later on and
wanted to at least mention it.

I am totally happy either way!

Best regards,
Sönke

On Wed, 18 Dec 2019 at 11:20, Mickael Maison <mi...@gmail.com> wrote:
>
> Thanks Sönke for the feedback.
>
> I debated this point quite a bit before deciding to base creation
> around "min.insync.replicas".
>
> For me, the goal of this KIP is to enable administrators to provide
> higher availability. In a 3 node cluster configured for high
> availability (3 replicas, 2 min ISR), by enabling this feature,
> clusters should be fully usable even when 1 broker is down. This
> should cover all "normal" maintenance operations like a rolling
> restart or just the recovery of a broker.
>
> At the moment, when creating a topic/partition, the assumption is that
> the resource will be fully functioning. This KIP does not change this
> assumption. If this is something someone wants, I think it should be
> handled in a different KIP that targets that use case. By relying on
> "min.insync.replicas", we don't break any assumptions the user has and
> this should be fully transparent from the user point of view.
>
> About "min.insync.replicas", one caveat that is not explicit in the
> KIP is that it's currently possible to create topics with less
> replicas than this settings. For that reason, I think the
> implementation will actually rely on min(replicas, min-isr) instead of
> simply min.insync.replicas. I have updated the KIP to explicitly
> mention this point.
>
> I hope that answers your question, let me know.
> Thanks
>
>
> On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> <so...@opencore.com.invalid> wrote:
> >
> > Hi Michael,
> >
> > that sounds like a useful addition! I can't help but wonder whether by
> > leaving in the restriction that "min.insync.replicas" has to be
> > satisfied we'll be back here in a years time because someone has a
> > scenario where he or she wants to go below that :)
> > I don't have a strong opinion either way to be honest, just a random
> > thought when reading the KIP.
> >
> > Best regards,
> > Sönke
> >
> > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <ry...@gmail.com> wrote:
> > >
> > > +1 non-binding, thx
> > >
> > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <mi...@gmail.com>
> > > wrote:
> > >
> > > > Bumping this thread, I've not seen any votes or feedback.
> > > >
> > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > <mi...@gmail.com> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start a vote on KIP-409: Allow creating under-replicated
> > > > > topics and partitions
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > >
> > > > > Thanks
> > > >
> >
> >
> >
> > --
> > Sönke Liebau
> > Partner
> > Tel. +49 179 7940878
> > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany



-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Mickael Maison <mi...@gmail.com>.

Thanks Sönke for the feedback.

I debated this point quite a bit before deciding to base creation
around "min.insync.replicas".

For me, the goal of this KIP is to enable administrators to provide
higher availability. In a 3 node cluster configured for high
availability (3 replicas, 2 min ISR), by enabling this feature,
clusters should be fully usable even when 1 broker is down. This
should cover all "normal" maintenance operations like a rolling
restart or just the recovery of a broker.

At the moment, when creating a topic/partition, the assumption is that
the resource will be fully functioning. This KIP does not change this
assumption. If this is something someone wants, I think it should be
handled in a different KIP that targets that use case. By relying on
"min.insync.replicas", we don't break any assumptions the user has and
this should be fully transparent from the user point of view.

About "min.insync.replicas", one caveat that is not explicit in the
KIP is that it's currently possible to create topics with less
replicas than this settings. For that reason, I think the
implementation will actually rely on min(replicas, min-isr) instead of
simply min.insync.replicas. I have updated the KIP to explicitly
mention this point.

I hope that answers your question, let me know.
Thanks

On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
<so...@opencore.com.invalid> wrote:
>
> Hi Michael,
>
> that sounds like a useful addition! I can't help but wonder whether by
> leaving in the restriction that "min.insync.replicas" has to be
> satisfied we'll be back here in a years time because someone has a
> scenario where he or she wants to go below that :)
> I don't have a strong opinion either way to be honest, just a random
> thought when reading the KIP.
>
> Best regards,
> Sönke
>
> On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <ry...@gmail.com> wrote:
> >
> > +1 non-binding, thx
> >
> > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> >
> > > Bumping this thread, I've not seen any votes or feedback.
> > >
> > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > <mi...@gmail.com> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I'd like to start a vote on KIP-409: Allow creating under-replicated
> > > > topics and partitions
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > >
> > > > Thanks
> > >
>
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Sönke Liebau <so...@opencore.com.INVALID>.

Hi Michael,

that sounds like a useful addition! I can't help but wonder whether by
leaving in the restriction that "min.insync.replicas" has to be
satisfied we'll be back here in a years time because someone has a
scenario where he or she wants to go below that :)
I don't have a strong opinion either way to be honest, just a random
thought when reading the KIP.

Best regards,
Sönke

On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <ry...@gmail.com> wrote:
>
> +1 non-binding, thx
>
> On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Bumping this thread, I've not seen any votes or feedback.
> >
> > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > <mi...@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to start a vote on KIP-409: Allow creating under-replicated
> > > topics and partitions
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > >
> > > Thanks
> >

-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Ryanne Dolan <ry...@gmail.com>.

+1 non-binding, thx

On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <mi...@gmail.com>
wrote:

> Bumping this thread, I've not seen any votes or feedback.
>
> On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> <mi...@gmail.com> wrote:
> >
> > Hi all,
> >
> > I'd like to start a vote on KIP-409: Allow creating under-replicated
> > topics and partitions
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> >
> > Thanks
>

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Posted by Mickael Maison <mi...@gmail.com>.

Bumping this thread, I've not seen any votes or feedback.

On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
<mi...@gmail.com> wrote:
>
> Hi all,
>
> I'd like to start a vote on KIP-409: Allow creating under-replicated
> topics and partitions
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
>
> Thanks