You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Satish Bellapu <sa...@gmail.com> on 2019/11/22 08:21:20 UTC
[DISCUSS] KIP-548 Add Option to enforce rack-aware custom partition reassignment execution
Hi All,
This [KIP-548] is basically extending the capabilities of "kafka-reassign-partitions" tool by adding rack-aware verification option when used along with custom or manually generated reassignment planner with --execute scenario.
@sbellapu.
Re: [DISCUSS] KIP-548 Add Option to enforce rack-aware custom partition reassignment execution
Posted by Satish Bellapu <sa...@gmail.com>.
Here is the link for the [KIP-548] https://cwiki.apache.org/confluence/display/KAFKA/KIP-548+Add+Option+to+enforce+rack-aware+custom+partition+reassignment+execution
On 2019/11/22 10:37:07, Stanislav Kozlovski <st...@confluent.io> wrote:
> Hello Satish,
>
> Could you provide a link to the KIP? I am unable to find it in the KIP
> parent page
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>
> Thanks,
> Stanislav
>
> On Fri, Nov 22, 2019 at 8:21 AM Satish Bellapu <sa...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > This [KIP-548] is basically extending the capabilities of
> > "kafka-reassign-partitions" tool by adding rack-aware verification option
> > when used along with custom or manually generated reassignment planner with
> > --execute scenario.
> >
> > @sbellapu.
> >
>
>
> --
> Best,
> Stanislav
>
Re: [DISCUSS] KIP-548 Add Option to enforce rack-aware custom partition reassignment execution
Posted by Satish Bellapu <sa...@gmail.com>.
Hi Viktor,
-1, @Vahid already provided the info.
-2, Moved the KIP to the parent path instead of KIP-36, thanks for finding out.
--sbellapu
On 2019/11/22 10:57:35, Viktor Somogyi-Vass <vi...@gmail.com> wrote:
> Hi Satish,
>
> Couple of questions/suggestions:
> 1. You say that when you execute the planned reassignment then it would
> throw an error if the generated reassignment doesn't comply with the
> rack-aware requirement. Opposed to this: why don't you have the --generate
> option to generate a rack-aware reassignment plan? This way users won't
> have to do the extra round.
> 2. Please move your KIP under
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals ,
> people will have a hard time finding it if it's under KIP-36.
> (@Stan fyi:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-548+Add+Option+to+enforce+rack-aware+custom+partition+reassignment+execution
> )
>
> Thanks,
> Viktor
>
> On Fri, Nov 22, 2019 at 11:37 AM Stanislav Kozlovski <st...@confluent.io>
> wrote:
>
> > Hello Satish,
> >
> > Could you provide a link to the KIP? I am unable to find it in the KIP
> > parent page
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >
> > Thanks,
> > Stanislav
> >
> > On Fri, Nov 22, 2019 at 8:21 AM Satish Bellapu <sa...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > This [KIP-548] is basically extending the capabilities of
> > > "kafka-reassign-partitions" tool by adding rack-aware verification option
> > > when used along with custom or manually generated reassignment planner
> > with
> > > --execute scenario.
> > >
> > > @sbellapu.
> > >
> >
> >
> > --
> > Best,
> > Stanislav
> >
>
Re: [DISCUSS] KIP-548 Add Option to enforce rack-aware custom partition reassignment execution
Posted by Satish Bellapu <sa...@gmail.com>.
Thanks for the comments @Vahid,
Yes i agree we can always introduce the new option, but what i was think is, as this is a ad-hoc tool and whoever is using the new version of the tool always will have the info to correct the usage, and also this will keep the tool in-sync with the current existing usage of other commands (at par with --generate).
If we deviate and start introduce new options then we might get into maintaining multiple options which are not align with the other commands so eventually that creates lot of unaligned options to the tool.
I am not sure how much effect it will bring when it's not backward compatible when using the tools under new release versions.
--Satish Bellapu.
On 2019/11/22 16:32:08, Vahid Hashemian <va...@gmail.com> wrote:
> Thanks Satish for drafting the KIP. Looks good overall. I would suggest
> emphasizing on the default option for --disable-rack-aware option when used
> with --execute option.
> Also, it would be great to also emphasize that the new format for
> --disable-rack-aware (which now takes a true/false value) would not impact
> the existing usages (e.g. with --generate option) that did not require a
> value for the option.
>
> Victor, to answer your first question, in my experience the assignment json
> file is not always created by the same command (through --generate option):
>
> - Sometimes when a broker is not healthy we manually update the existing
> assignment to change partition replicas to reduce load on the degraded
> broker.
> - In generating full partition assignment plan we also want use some
> custom assignment strategy to have more control over partition placements
> and do not use the default strategy used by Kafka.
>
> In these scenarios, it would be very helpful to have the option of
> enforcing rack awareness with the command's --execute option.
>
> Regards,
> --Vahid
>
> On Fri, Nov 22, 2019 at 2:57 AM Viktor Somogyi-Vass <vi...@gmail.com>
> wrote:
>
> > Hi Satish,
> >
> > Couple of questions/suggestions:
> > 1. You say that when you execute the planned reassignment then it would
> > throw an error if the generated reassignment doesn't comply with the
> > rack-aware requirement. Opposed to this: why don't you have the --generate
> > option to generate a rack-aware reassignment plan? This way users won't
> > have to do the extra round.
> > 2. Please move your KIP under
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > ,
> > people will have a hard time finding it if it's under KIP-36.
> > (@Stan fyi:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-548+Add+Option+to+enforce+rack-aware+custom+partition+reassignment+execution
> > )
> >
> > Thanks,
> > Viktor
> >
> > On Fri, Nov 22, 2019 at 11:37 AM Stanislav Kozlovski <
> > stanislav@confluent.io>
> > wrote:
> >
> > > Hello Satish,
> > >
> > > Could you provide a link to the KIP? I am unable to find it in the KIP
> > > parent page
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >
> > > Thanks,
> > > Stanislav
> > >
> > > On Fri, Nov 22, 2019 at 8:21 AM Satish Bellapu <sa...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > This [KIP-548] is basically extending the capabilities of
> > > > "kafka-reassign-partitions" tool by adding rack-aware verification
> > option
> > > > when used along with custom or manually generated reassignment planner
> > > with
> > > > --execute scenario.
> > > >
> > > > @sbellapu.
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> >
>
>
> --
>
> Thanks!
> --Vahid
>
Re: [DISCUSS] KIP-548 Add Option to enforce rack-aware custom
partition reassignment execution
Posted by Vahid Hashemian <va...@gmail.com>.
Hi Satish,
Thanks for the update. What you suggest in the KIP is ideal, the only issue
is the --execute option will not be backward compatible; so the same
command that used to work before, may no longer work if this suggestion is
implemented. One option for keeping backward compatibility would be using a
new option, such as --enable-rack-aware that either works only with
--execute, or works with other options too. This would not necessary be the
best option, but at least it would not be a breaking change.
Let's see if others have better ideas.
Thanks,
--Vahid
On Fri, Nov 22, 2019 at 5:59 PM Satish Bellapu <sa...@gmail.com>
wrote:
> Hi Vahid,
> After re-thinking on this, i have following updates on the KIP, with
> aligning to the other options on ReassignPartitionsCommand,
>
> The --execute command by default take rack awareness into consideration,
> and if the custom generated reassignment planner has conflicts along with
> the racks then it will throw the error msg with appropriate reason and
> conflict of partitions along with the racks info. The users need to
> explicitly choose the option --disable-rack-aware if they want to ignore
> the rack awareness.
>
> By this change the usage of options in --execute command will be aligned
> with --generate option, the rack awareness will be consider by default both
> for --generate as well as --execute unless explicitly set to
> --disable-rack-aware.
>
> Let me know whats your thoughts on the same.
>
> --sbellapu
>
> On 2019/11/22 16:32:08, Vahid Hashemian <va...@gmail.com>
> wrote:
> > Thanks Satish for drafting the KIP. Looks good overall. I would suggest
> > emphasizing on the default option for --disable-rack-aware option when
> used
> > with --execute option.
> > Also, it would be great to also emphasize that the new format for
> > --disable-rack-aware (which now takes a true/false value) would not
> impact
> > the existing usages (e.g. with --generate option) that did not require a
> > value for the option.
> >
> > Victor, to answer your first question, in my experience the assignment
> json
> > file is not always created by the same command (through --generate
> option):
> >
> > - Sometimes when a broker is not healthy we manually update the
> existing
> > assignment to change partition replicas to reduce load on the degraded
> > broker.
> > - In generating full partition assignment plan we also want use some
> > custom assignment strategy to have more control over partition
> placements
> > and do not use the default strategy used by Kafka.
> >
> > In these scenarios, it would be very helpful to have the option of
> > enforcing rack awareness with the command's --execute option.
> >
> > Regards,
> > --Vahid
> >
> > On Fri, Nov 22, 2019 at 2:57 AM Viktor Somogyi-Vass <
> viktorsomogyi@gmail.com>
> > wrote:
> >
> > > Hi Satish,
> > >
> > > Couple of questions/suggestions:
> > > 1. You say that when you execute the planned reassignment then it would
> > > throw an error if the generated reassignment doesn't comply with the
> > > rack-aware requirement. Opposed to this: why don't you have the
> --generate
> > > option to generate a rack-aware reassignment plan? This way users won't
> > > have to do the extra round.
> > > 2. Please move your KIP under
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > ,
> > > people will have a hard time finding it if it's under KIP-36.
> > > (@Stan fyi:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-548+Add+Option+to+enforce+rack-aware+custom+partition+reassignment+execution
> > > )
> > >
> > > Thanks,
> > > Viktor
> > >
> > > On Fri, Nov 22, 2019 at 11:37 AM Stanislav Kozlovski <
> > > stanislav@confluent.io>
> > > wrote:
> > >
> > > > Hello Satish,
> > > >
> > > > Could you provide a link to the KIP? I am unable to find it in the
> KIP
> > > > parent page
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > >
> > > > Thanks,
> > > > Stanislav
> > > >
> > > > On Fri, Nov 22, 2019 at 8:21 AM Satish Bellapu <
> satishbabumsc@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > This [KIP-548] is basically extending the capabilities of
> > > > > "kafka-reassign-partitions" tool by adding rack-aware verification
> > > option
> > > > > when used along with custom or manually generated reassignment
> planner
> > > > with
> > > > > --execute scenario.
> > > > >
> > > > > @sbellapu.
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Stanislav
> > > >
> > >
> >
> >
> > --
> >
> > Thanks!
> > --Vahid
> >
>
--
Thanks!
--Vahid
Re: [DISCUSS] KIP-548 Add Option to enforce rack-aware custom partition reassignment execution
Posted by Satish Bellapu <sa...@gmail.com>.
Hi Vahid,
After re-thinking on this, i have following updates on the KIP, with aligning to the other options on ReassignPartitionsCommand,
The --execute command by default take rack awareness into consideration, and if the custom generated reassignment planner has conflicts along with the racks then it will throw the error msg with appropriate reason and conflict of partitions along with the racks info. The users need to explicitly choose the option --disable-rack-aware if they want to ignore the rack awareness.
By this change the usage of options in --execute command will be aligned with --generate option, the rack awareness will be consider by default both for --generate as well as --execute unless explicitly set to --disable-rack-aware.
Let me know whats your thoughts on the same.
--sbellapu
On 2019/11/22 16:32:08, Vahid Hashemian <va...@gmail.com> wrote:
> Thanks Satish for drafting the KIP. Looks good overall. I would suggest
> emphasizing on the default option for --disable-rack-aware option when used
> with --execute option.
> Also, it would be great to also emphasize that the new format for
> --disable-rack-aware (which now takes a true/false value) would not impact
> the existing usages (e.g. with --generate option) that did not require a
> value for the option.
>
> Victor, to answer your first question, in my experience the assignment json
> file is not always created by the same command (through --generate option):
>
> - Sometimes when a broker is not healthy we manually update the existing
> assignment to change partition replicas to reduce load on the degraded
> broker.
> - In generating full partition assignment plan we also want use some
> custom assignment strategy to have more control over partition placements
> and do not use the default strategy used by Kafka.
>
> In these scenarios, it would be very helpful to have the option of
> enforcing rack awareness with the command's --execute option.
>
> Regards,
> --Vahid
>
> On Fri, Nov 22, 2019 at 2:57 AM Viktor Somogyi-Vass <vi...@gmail.com>
> wrote:
>
> > Hi Satish,
> >
> > Couple of questions/suggestions:
> > 1. You say that when you execute the planned reassignment then it would
> > throw an error if the generated reassignment doesn't comply with the
> > rack-aware requirement. Opposed to this: why don't you have the --generate
> > option to generate a rack-aware reassignment plan? This way users won't
> > have to do the extra round.
> > 2. Please move your KIP under
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > ,
> > people will have a hard time finding it if it's under KIP-36.
> > (@Stan fyi:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-548+Add+Option+to+enforce+rack-aware+custom+partition+reassignment+execution
> > )
> >
> > Thanks,
> > Viktor
> >
> > On Fri, Nov 22, 2019 at 11:37 AM Stanislav Kozlovski <
> > stanislav@confluent.io>
> > wrote:
> >
> > > Hello Satish,
> > >
> > > Could you provide a link to the KIP? I am unable to find it in the KIP
> > > parent page
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >
> > > Thanks,
> > > Stanislav
> > >
> > > On Fri, Nov 22, 2019 at 8:21 AM Satish Bellapu <sa...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > This [KIP-548] is basically extending the capabilities of
> > > > "kafka-reassign-partitions" tool by adding rack-aware verification
> > option
> > > > when used along with custom or manually generated reassignment planner
> > > with
> > > > --execute scenario.
> > > >
> > > > @sbellapu.
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> >
>
>
> --
>
> Thanks!
> --Vahid
>
Re: [DISCUSS] KIP-548 Add Option to enforce rack-aware custom
partition reassignment execution
Posted by Vahid Hashemian <va...@gmail.com>.
Thanks Satish for drafting the KIP. Looks good overall. I would suggest
emphasizing on the default option for --disable-rack-aware option when used
with --execute option.
Also, it would be great to also emphasize that the new format for
--disable-rack-aware (which now takes a true/false value) would not impact
the existing usages (e.g. with --generate option) that did not require a
value for the option.
Victor, to answer your first question, in my experience the assignment json
file is not always created by the same command (through --generate option):
- Sometimes when a broker is not healthy we manually update the existing
assignment to change partition replicas to reduce load on the degraded
broker.
- In generating full partition assignment plan we also want use some
custom assignment strategy to have more control over partition placements
and do not use the default strategy used by Kafka.
In these scenarios, it would be very helpful to have the option of
enforcing rack awareness with the command's --execute option.
Regards,
--Vahid
On Fri, Nov 22, 2019 at 2:57 AM Viktor Somogyi-Vass <vi...@gmail.com>
wrote:
> Hi Satish,
>
> Couple of questions/suggestions:
> 1. You say that when you execute the planned reassignment then it would
> throw an error if the generated reassignment doesn't comply with the
> rack-aware requirement. Opposed to this: why don't you have the --generate
> option to generate a rack-aware reassignment plan? This way users won't
> have to do the extra round.
> 2. Please move your KIP under
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> ,
> people will have a hard time finding it if it's under KIP-36.
> (@Stan fyi:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-548+Add+Option+to+enforce+rack-aware+custom+partition+reassignment+execution
> )
>
> Thanks,
> Viktor
>
> On Fri, Nov 22, 2019 at 11:37 AM Stanislav Kozlovski <
> stanislav@confluent.io>
> wrote:
>
> > Hello Satish,
> >
> > Could you provide a link to the KIP? I am unable to find it in the KIP
> > parent page
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >
> > Thanks,
> > Stanislav
> >
> > On Fri, Nov 22, 2019 at 8:21 AM Satish Bellapu <sa...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > This [KIP-548] is basically extending the capabilities of
> > > "kafka-reassign-partitions" tool by adding rack-aware verification
> option
> > > when used along with custom or manually generated reassignment planner
> > with
> > > --execute scenario.
> > >
> > > @sbellapu.
> > >
> >
> >
> > --
> > Best,
> > Stanislav
> >
>
--
Thanks!
--Vahid
Re: [DISCUSS] KIP-548 Add Option to enforce rack-aware custom
partition reassignment execution
Posted by Viktor Somogyi-Vass <vi...@gmail.com>.
Hi Satish,
Couple of questions/suggestions:
1. You say that when you execute the planned reassignment then it would
throw an error if the generated reassignment doesn't comply with the
rack-aware requirement. Opposed to this: why don't you have the --generate
option to generate a rack-aware reassignment plan? This way users won't
have to do the extra round.
2. Please move your KIP under
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals ,
people will have a hard time finding it if it's under KIP-36.
(@Stan fyi:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-548+Add+Option+to+enforce+rack-aware+custom+partition+reassignment+execution
)
Thanks,
Viktor
On Fri, Nov 22, 2019 at 11:37 AM Stanislav Kozlovski <st...@confluent.io>
wrote:
> Hello Satish,
>
> Could you provide a link to the KIP? I am unable to find it in the KIP
> parent page
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>
> Thanks,
> Stanislav
>
> On Fri, Nov 22, 2019 at 8:21 AM Satish Bellapu <sa...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > This [KIP-548] is basically extending the capabilities of
> > "kafka-reassign-partitions" tool by adding rack-aware verification option
> > when used along with custom or manually generated reassignment planner
> with
> > --execute scenario.
> >
> > @sbellapu.
> >
>
>
> --
> Best,
> Stanislav
>
Re: [DISCUSS] KIP-548 Add Option to enforce rack-aware custom
partition reassignment execution
Posted by Stanislav Kozlovski <st...@confluent.io>.
Hello Satish,
Could you provide a link to the KIP? I am unable to find it in the KIP
parent page
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
Thanks,
Stanislav
On Fri, Nov 22, 2019 at 8:21 AM Satish Bellapu <sa...@gmail.com>
wrote:
> Hi All,
>
> This [KIP-548] is basically extending the capabilities of
> "kafka-reassign-partitions" tool by adding rack-aware verification option
> when used along with custom or manually generated reassignment planner with
> --execute scenario.
>
> @sbellapu.
>
--
Best,
Stanislav