You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Kuan Po Tseng <br...@apache.org> on 2025/02/05 14:46:08 UTC
Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments
Hi all,
After discussing with Stanislav Kozlovski, I plan to continue implementing this KIP.
However, I have a question regarding the Validation section.
It says, "If the targetReplicaSet of the reassignment differs from the current replica set
of the partition, an error is thrown."
I’m wondering if the word “differs” here refers to:
(1) Only the targetReplicaSet size differing from the current replica set, or
(2) The targetReplicaSet as a whole differing from the current replica set.
I assume it’s (2) since the KIP aims to avoid a situation where, during alterPartitionReassignments, the replication factor (RF) could end up
larger than the original RF. However, I would like to confirm if
my understanding is correct.
Thank you!
--
Best,
Kuan-Po
On 2022/07/28 08:59:18 Stanislav Kozlovski wrote:
> Hey all,
>
> I'd like to start a discussion on a proposal to help API users from
> inadvertently increasing the replication factor of a topic through
> the alter partition reassignments API. The KIP describes two fairly
> easy-to-hit race conditions in which this can happen.
>
> The KIP itself is pretty simple, yet has a couple of alternatives that can
> help solve the same problem. I would appreciate thoughts from the community
> on how you think we should proceed, and whether the proposal makes sense in
> the first place.
>
> Thanks!
>
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
> JIRA: https://issues.apache.org/jira/browse/KAFKA-14121
>
> --
> Best,
> Stanislav
>
Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments
Posted by Kuan Po Tseng <br...@apache.org>.
Hi everyone,
Just a quick update—I’ve made some adjustments to the KIP:
- Changed allowReplicationFactorChange from Boolean to boolean.
- Adopted the fluent pattern in AlterPartitionReassignmentsOptions.
Also, the PR has been merged into trunk, targeting version 4.1.0.
Thanks again for your support!
Best,
Kuan-Po Tseng
On 2025/02/06 18:42:29 Chia-Ping Tsai wrote:
> hi Stan
>
> thanks for sharing the details!
>
> to Kuan-Po
>
> Could you please tweak the KIP-860 according to Stan's response :)
>
> Best,
> Chia-Ping
>
> Stanislav Kozlovski <st...@apache.org> 於 2025年2月7日 週五 上午2:32寫道：
>
> > Hey Kuan-Po!
> >
> > First of all - thanks so much for volunteering to take the implementation
> > of this KIP!
> >
> > When I initially proposed it, I had planned with my team in Confluent to
> > have somebody else implement it as a chance to get more experience with the
> > open-source project. Sadly, we never got to it.
> >
> > To answer your question:
> >
> > The idea of this KIP is to prevent the replication factor from changing,
> > so it's ultimately only the size that should get checked. As mentioned - if
> > the partition is NOT reassigning, then we check against
> > `targetReplicaSet.size != currentReplicaSet.size`. If the partition IS
> > reasigning, then we check `currentReassignmentTargetReplicaSet.size !=
> > newTargetReplicaSet.size`
> >
> > Best,
> > Stan
> >
> > On 2025/02/05 14:46:08 Kuan Po Tseng wrote:
> > > Hi all,
> > >
> > > After discussing with Stanislav Kozlovski, I plan to continue
> > implementing this KIP.
> > > However, I have a question regarding the Validation section.
> > >
> > > It says, "If the targetReplicaSet of the reassignment differs from the
> > current replica set
> > > of the partition, an error is thrown."
> > >
> > > I’m wondering if the word “differs” here refers to:
> > > (1) Only the targetReplicaSet size differing from the current replica
> > set, or
> > > (2) The targetReplicaSet as a whole differing from the current replica
> > set.
> > >
> > > I assume it’s (2) since the KIP aims to avoid a situation where, during
> > alterPartitionReassignments, the replication factor (RF) could end up
> > > larger than the original RF. However, I would like to confirm if
> > > my understanding is correct.
> > >
> > > Thank you!
> > >
> > > --
> > > Best,
> > > Kuan-Po
> > >
> > > On 2022/07/28 08:59:18 Stanislav Kozlovski wrote:
> > > > Hey all,
> > > >
> > > > I'd like to start a discussion on a proposal to help API users from
> > > > inadvertently increasing the replication factor of a topic through
> > > > the alter partition reassignments API. The KIP describes two fairly
> > > > easy-to-hit race conditions in which this can happen.
> > > >
> > > > The KIP itself is pretty simple, yet has a couple of alternatives that
> > can
> > > > help solve the same problem. I would appreciate thoughts from the
> > community
> > > > on how you think we should proceed, and whether the proposal makes
> > sense in
> > > > the first place.
> > > >
> > > > Thanks!
> > > >
> > > > KIP:
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
> > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-14121
> > > >
> > > > --
> > > > Best,
> > > > Stanislav
> > > >
> > >
> >
>
Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments
Posted by Kuan Po Tseng <br...@apache.org>.
Hi everyone,
Thanks to Stanislav for answering the question!
In the beginning I thought
I’ve also updated the KIP based on the discussion above.
Best,
Kuan-Po
On 2025/02/06 18:42:29 Chia-Ping Tsai wrote:
> hi Stan
>
> thanks for sharing the details!
>
> to Kuan-Po
>
> Could you please tweak the KIP-860 according to Stan's response :)
>
> Best,
> Chia-Ping
>
> Stanislav Kozlovski <st...@apache.org> 於 2025年2月7日 週五 上午2:32寫道：
>
> > Hey Kuan-Po!
> >
> > First of all - thanks so much for volunteering to take the implementation
> > of this KIP!
> >
> > When I initially proposed it, I had planned with my team in Confluent to
> > have somebody else implement it as a chance to get more experience with the
> > open-source project. Sadly, we never got to it.
> >
> > To answer your question:
> >
> > The idea of this KIP is to prevent the replication factor from changing,
> > so it's ultimately only the size that should get checked. As mentioned - if
> > the partition is NOT reassigning, then we check against
> > `targetReplicaSet.size != currentReplicaSet.size`. If the partition IS
> > reasigning, then we check `currentReassignmentTargetReplicaSet.size !=
> > newTargetReplicaSet.size`
> >
> > Best,
> > Stan
> >
> > On 2025/02/05 14:46:08 Kuan Po Tseng wrote:
> > > Hi all,
> > >
> > > After discussing with Stanislav Kozlovski, I plan to continue
> > implementing this KIP.
> > > However, I have a question regarding the Validation section.
> > >
> > > It says, "If the targetReplicaSet of the reassignment differs from the
> > current replica set
> > > of the partition, an error is thrown."
> > >
> > > I’m wondering if the word “differs” here refers to:
> > > (1) Only the targetReplicaSet size differing from the current replica
> > set, or
> > > (2) The targetReplicaSet as a whole differing from the current replica
> > set.
> > >
> > > I assume it’s (2) since the KIP aims to avoid a situation where, during
> > alterPartitionReassignments, the replication factor (RF) could end up
> > > larger than the original RF. However, I would like to confirm if
> > > my understanding is correct.
> > >
> > > Thank you!
> > >
> > > --
> > > Best,
> > > Kuan-Po
> > >
> > > On 2022/07/28 08:59:18 Stanislav Kozlovski wrote:
> > > > Hey all,
> > > >
> > > > I'd like to start a discussion on a proposal to help API users from
> > > > inadvertently increasing the replication factor of a topic through
> > > > the alter partition reassignments API. The KIP describes two fairly
> > > > easy-to-hit race conditions in which this can happen.
> > > >
> > > > The KIP itself is pretty simple, yet has a couple of alternatives that
> > can
> > > > help solve the same problem. I would appreciate thoughts from the
> > community
> > > > on how you think we should proceed, and whether the proposal makes
> > sense in
> > > > the first place.
> > > >
> > > > Thanks!
> > > >
> > > > KIP:
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
> > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-14121
> > > >
> > > > --
> > > > Best,
> > > > Stanislav
> > > >
> > >
> >
>
Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments
Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi Stan
thanks for sharing the details!
to Kuan-Po
Could you please tweak the KIP-860 according to Stan's response :)
Best,
Chia-Ping
Stanislav Kozlovski <st...@apache.org> 於 2025年2月7日 週五 上午2:32寫道：
> Hey Kuan-Po!
>
> First of all - thanks so much for volunteering to take the implementation
> of this KIP!
>
> When I initially proposed it, I had planned with my team in Confluent to
> have somebody else implement it as a chance to get more experience with the
> open-source project. Sadly, we never got to it.
>
> To answer your question:
>
> The idea of this KIP is to prevent the replication factor from changing,
> so it's ultimately only the size that should get checked. As mentioned - if
> the partition is NOT reassigning, then we check against
> `targetReplicaSet.size != currentReplicaSet.size`. If the partition IS
> reasigning, then we check `currentReassignmentTargetReplicaSet.size !=
> newTargetReplicaSet.size`
>
> Best,
> Stan
>
> On 2025/02/05 14:46:08 Kuan Po Tseng wrote:
> > Hi all,
> >
> > After discussing with Stanislav Kozlovski, I plan to continue
> implementing this KIP.
> > However, I have a question regarding the Validation section.
> >
> > It says, "If the targetReplicaSet of the reassignment differs from the
> current replica set
> > of the partition, an error is thrown."
> >
> > I’m wondering if the word “differs” here refers to:
> > (1) Only the targetReplicaSet size differing from the current replica
> set, or
> > (2) The targetReplicaSet as a whole differing from the current replica
> set.
> >
> > I assume it’s (2) since the KIP aims to avoid a situation where, during
> alterPartitionReassignments, the replication factor (RF) could end up
> > larger than the original RF. However, I would like to confirm if
> > my understanding is correct.
> >
> > Thank you!
> >
> > --
> > Best,
> > Kuan-Po
> >
> > On 2022/07/28 08:59:18 Stanislav Kozlovski wrote:
> > > Hey all,
> > >
> > > I'd like to start a discussion on a proposal to help API users from
> > > inadvertently increasing the replication factor of a topic through
> > > the alter partition reassignments API. The KIP describes two fairly
> > > easy-to-hit race conditions in which this can happen.
> > >
> > > The KIP itself is pretty simple, yet has a couple of alternatives that
> can
> > > help solve the same problem. I would appreciate thoughts from the
> community
> > > on how you think we should proceed, and whether the proposal makes
> sense in
> > > the first place.
> > >
> > > Thanks!
> > >
> > > KIP:
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-14121
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> >
>
Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments
Posted by Stanislav Kozlovski <st...@apache.org>.
Hey Kuan-Po!
First of all - thanks so much for volunteering to take the implementation of this KIP!
When I initially proposed it, I had planned with my team in Confluent to have somebody else implement it as a chance to get more experience with the open-source project. Sadly, we never got to it.
To answer your question:
The idea of this KIP is to prevent the replication factor from changing, so it's ultimately only the size that should get checked. As mentioned - if the partition is NOT reassigning, then we check against `targetReplicaSet.size != currentReplicaSet.size`. If the partition IS reasigning, then we check `currentReassignmentTargetReplicaSet.size != newTargetReplicaSet.size`
Best,
Stan
On 2025/02/05 14:46:08 Kuan Po Tseng wrote:
> Hi all,
>
> After discussing with Stanislav Kozlovski, I plan to continue implementing this KIP.
> However, I have a question regarding the Validation section.
>
> It says, "If the targetReplicaSet of the reassignment differs from the current replica set
> of the partition, an error is thrown."
>
> I’m wondering if the word “differs” here refers to:
> (1) Only the targetReplicaSet size differing from the current replica set, or
> (2) The targetReplicaSet as a whole differing from the current replica set.
>
> I assume it’s (2) since the KIP aims to avoid a situation where, during alterPartitionReassignments, the replication factor (RF) could end up
> larger than the original RF. However, I would like to confirm if
> my understanding is correct.
>
> Thank you!
>
> --
> Best,
> Kuan-Po
>
> On 2022/07/28 08:59:18 Stanislav Kozlovski wrote:
> > Hey all,
> >
> > I'd like to start a discussion on a proposal to help API users from
> > inadvertently increasing the replication factor of a topic through
> > the alter partition reassignments API. The KIP describes two fairly
> > easy-to-hit race conditions in which this can happen.
> >
> > The KIP itself is pretty simple, yet has a couple of alternatives that can
> > help solve the same problem. I would appreciate thoughts from the community
> > on how you think we should proceed, and whether the proposal makes sense in
> > the first place.
> >
> > Thanks!
> >
> > KIP:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
> > JIRA: https://issues.apache.org/jira/browse/KAFKA-14121
> >
> > --
> > Best,
> > Stanislav
> >
>