You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Tom Bentley <t....@gmail.com> on 2017/09/08 16:42:05 UTC
[VOTE] KIP-195: AdminClient.createPartitions()
I would like to start the vote on KIP-195 which adds an AdminClient API for
increasing the number of partitions of a topic. The details are here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.createPartitions
Cheers,
Tom
Re: [VOTE] KIP-195: AdminClient.createPartitions()
Posted by Tom Bentley <t....@gmail.com>.
The vote has passed. with three binding +1s (Jun, Ismael and Jason), no -1s
or +0s.
Thanks to those who commented and voted.
On 13 September 2017 at 23:22, Jason Gustafson <ja...@confluent.io> wrote:
> +1. Thanks for the KIP. Just one minor clarification: when no assignment is
> specified by the user, what will be sent in the protocol? A null assignment
> array? Probably worth mentioning this case explicitly in the KIP.
>
> Thanks,
> Jason
>
> On Wed, Sep 13, 2017 at 7:53 AM, Tom Bentley <t....@gmail.com>
> wrote:
>
> > This KIP currently has 2 binding +1 votes. Today is the deadline for KIPs
> > to be added to Kafka 1.0.0. So if anyone else would like to see this
> > feature in 1.0.0 they will need to vote by the end of the day.
> >
> > If there are insufficient votes for the KIP to be adopted today then I
> will
> > keep the vote open for a while longer, but it won't be in 1.0.0 even if
> the
> > vote is eventually successful.
> >
> > Cheers,
> >
> > Tom
> >
> > On 13 September 2017 at 11:43, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Tom,
> > >
> > > Thanks for the KIP, +1 (binding) from me.
> > >
> > > Ismael
> > >
> > > On Fri, Sep 8, 2017 at 5:42 PM, Tom Bentley <t....@gmail.com>
> > wrote:
> > >
> > > > I would like to start the vote on KIP-195 which adds an AdminClient
> API
> > > for
> > > > increasing the number of partitions of a topic. The details are here:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 195%3A+AdminClient
> > > .
> > > > createPartitions
> > > >
> > > > Cheers,
> > > >
> > > > Tom
> > > >
> > >
> >
>
Re: [VOTE] KIP-195: AdminClient.createPartitions()
Posted by Jason Gustafson <ja...@confluent.io>.
+1. Thanks for the KIP. Just one minor clarification: when no assignment is
specified by the user, what will be sent in the protocol? A null assignment
array? Probably worth mentioning this case explicitly in the KIP.
Thanks,
Jason
On Wed, Sep 13, 2017 at 7:53 AM, Tom Bentley <t....@gmail.com> wrote:
> This KIP currently has 2 binding +1 votes. Today is the deadline for KIPs
> to be added to Kafka 1.0.0. So if anyone else would like to see this
> feature in 1.0.0 they will need to vote by the end of the day.
>
> If there are insufficient votes for the KIP to be adopted today then I will
> keep the vote open for a while longer, but it won't be in 1.0.0 even if the
> vote is eventually successful.
>
> Cheers,
>
> Tom
>
> On 13 September 2017 at 11:43, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Tom,
> >
> > Thanks for the KIP, +1 (binding) from me.
> >
> > Ismael
> >
> > On Fri, Sep 8, 2017 at 5:42 PM, Tom Bentley <t....@gmail.com>
> wrote:
> >
> > > I would like to start the vote on KIP-195 which adds an AdminClient API
> > for
> > > increasing the number of partitions of a topic. The details are here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 195%3A+AdminClient
> > .
> > > createPartitions
> > >
> > > Cheers,
> > >
> > > Tom
> > >
> >
>
Re: [VOTE] KIP-195: AdminClient.createPartitions()
Posted by Tom Bentley <t....@gmail.com>.
This KIP currently has 2 binding +1 votes. Today is the deadline for KIPs
to be added to Kafka 1.0.0. So if anyone else would like to see this
feature in 1.0.0 they will need to vote by the end of the day.
If there are insufficient votes for the KIP to be adopted today then I will
keep the vote open for a while longer, but it won't be in 1.0.0 even if the
vote is eventually successful.
Cheers,
Tom
On 13 September 2017 at 11:43, Ismael Juma <is...@juma.me.uk> wrote:
> Tom,
>
> Thanks for the KIP, +1 (binding) from me.
>
> Ismael
>
> On Fri, Sep 8, 2017 at 5:42 PM, Tom Bentley <t....@gmail.com> wrote:
>
> > I would like to start the vote on KIP-195 which adds an AdminClient API
> for
> > increasing the number of partitions of a topic. The details are here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient
> .
> > createPartitions
> >
> > Cheers,
> >
> > Tom
> >
>
Re: [VOTE] KIP-195: AdminClient.createPartitions()
Posted by Ismael Juma <is...@juma.me.uk>.
Tom,
Thanks for the KIP, +1 (binding) from me.
Ismael
On Fri, Sep 8, 2017 at 5:42 PM, Tom Bentley <t....@gmail.com> wrote:
> I would like to start the vote on KIP-195 which adds an AdminClient API for
> increasing the number of partitions of a topic. The details are here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.
> createPartitions
>
> Cheers,
>
> Tom
>
Re: [VOTE] KIP-195: AdminClient.createPartitions()
Posted by Jun Rao <ju...@confluent.io>.
Hi, Tom,
2. Thanks for the explanation. That makes sense and we can leave this as it
is.
Jun
On Tue, Sep 12, 2017 at 10:46 AM, Tom Bentley <t....@gmail.com> wrote:
> Hi Jun,
>
> Thanks for the comments.
>
> On 12 September 2017 at 18:15, Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Tom,
> >
> > Thanks for the KIP. +1. Just a couple of minor comments below.
> >
> > 1. The KIP has "INVALID_PARTITIONS (37) If the partition count was <= the
> > current partition count for the topic." We probably want to add one more
> > constraint: # of replicas in each new partition has to be the same as the
> > existing replication factor for the topic.
> >
>
> This is presently covered by the INVALID_REQUEST error code, but sure I can
> change it.
>
> 2. The KIP has the following.
> >
> > - REASSIGNMENT_IN_PROGRESS (new) If a partition reassignment is in
> > progress. It is necessary to prevent increasing partitions at the same
> > time so that we can be sure the partition has a meaningful replication
> > factor.
> >
> >
> > Currently, it's possible to increase the partition count while a
> > partition reassignment is in progress (adding partitions is much
> > cheaper than partition reassignment). On the other hand, if a topic is
> > being deleted, we will prevent adding new partitions. So, we probably
> > want to do the same in the KIP.
> >
>
> Increasing the partition count at the same time as a reassignment was
> covered in the [DISCUSS] thread. The problem is that during a reassignment
> there isn't a stable notion of replication factor. The brokers don't really
> know about replication factor at all, AFAICS. Currently when the partitions
> are increased we just use the replication factor inferred from partition 0.
> Ismael suggested to add this restriction to prevent this edge case.
>
> I don't see an error code specifically pertaining to topic actions on
> topics being deleted, so I guess INVALID_TOPIC_EXCEPTION would suffice for
> the deletion case.
>
> Thanks again,
>
> Tom
>
Re: [VOTE] KIP-195: AdminClient.createPartitions()
Posted by Tom Bentley <t....@gmail.com>.
Hi Jun,
Thanks for the comments.
On 12 September 2017 at 18:15, Jun Rao <ju...@confluent.io> wrote:
> Hi, Tom,
>
> Thanks for the KIP. +1. Just a couple of minor comments below.
>
> 1. The KIP has "INVALID_PARTITIONS (37) If the partition count was <= the
> current partition count for the topic." We probably want to add one more
> constraint: # of replicas in each new partition has to be the same as the
> existing replication factor for the topic.
>
This is presently covered by the INVALID_REQUEST error code, but sure I can
change it.
2. The KIP has the following.
>
> - REASSIGNMENT_IN_PROGRESS (new) If a partition reassignment is in
> progress. It is necessary to prevent increasing partitions at the same
> time so that we can be sure the partition has a meaningful replication
> factor.
>
>
> Currently, it's possible to increase the partition count while a
> partition reassignment is in progress (adding partitions is much
> cheaper than partition reassignment). On the other hand, if a topic is
> being deleted, we will prevent adding new partitions. So, we probably
> want to do the same in the KIP.
>
Increasing the partition count at the same time as a reassignment was
covered in the [DISCUSS] thread. The problem is that during a reassignment
there isn't a stable notion of replication factor. The brokers don't really
know about replication factor at all, AFAICS. Currently when the partitions
are increased we just use the replication factor inferred from partition 0.
Ismael suggested to add this restriction to prevent this edge case.
I don't see an error code specifically pertaining to topic actions on
topics being deleted, so I guess INVALID_TOPIC_EXCEPTION would suffice for
the deletion case.
Thanks again,
Tom
Re: [VOTE] KIP-195: AdminClient.createPartitions()
Posted by Jun Rao <ju...@confluent.io>.
Hi, Tom,
Thanks for the KIP. +1. Just a couple of minor comments below.
1. The KIP has "INVALID_PARTITIONS (37) If the partition count was <= the
current partition count for the topic." We probably want to add one more
constraint: # of replicas in each new partition has to be the same as the
existing replication factor for the topic.
2. The KIP has the following.
- REASSIGNMENT_IN_PROGRESS (new) If a partition reassignment is in
progress. It is necessary to prevent increasing partitions at the same
time so that we can be sure the partition has a meaningful replication
factor.
Currently, it's possible to increase the partition count while a
partition reassignment is in progress (adding partitions is much
cheaper than partition reassignment). On the other hand, if a topic is
being deleted, we will prevent adding new partitions. So, we probably
want to do the same in the KIP.
Jun
On Tue, Sep 12, 2017 at 9:35 AM, Tom Bentley <t....@gmail.com> wrote:
> Following additional comments from Ismael I have updated the KIP slightly
> to:
>
> * No longer apply the CreateTopicPolicy (a future KIP will address applying
> a policy to topic changes)
> * Clarify that the request must be sent to the controller and the response
> will only be sent once the changes are reflected in the metadata cache.
>
> Cheers,
>
> Tom
>
> On 8 September 2017 at 17:42, Tom Bentley <t....@gmail.com> wrote:
>
> > I would like to start the vote on KIP-195 which adds an AdminClient API
> > for increasing the number of partitions of a topic. The details are here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient
> .
> > createPartitions
> >
> > Cheers,
> >
> > Tom
> >
>
Re: [VOTE] KIP-195: AdminClient.createPartitions()
Posted by Tom Bentley <t....@gmail.com>.
Following additional comments from Ismael I have updated the KIP slightly
to:
* No longer apply the CreateTopicPolicy (a future KIP will address applying
a policy to topic changes)
* Clarify that the request must be sent to the controller and the response
will only be sent once the changes are reflected in the metadata cache.
Cheers,
Tom
On 8 September 2017 at 17:42, Tom Bentley <t....@gmail.com> wrote:
> I would like to start the vote on KIP-195 which adds an AdminClient API
> for increasing the number of partitions of a topic. The details are here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.
> createPartitions
>
> Cheers,
>
> Tom
>