You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Kuan Po Tseng <br...@gmail.com> on 2025/03/10 01:26:21 UTC
KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Hi everyone,
I'd like to start a discussion thread on *KIP-1141*, which proposes a
relatively small but useful change:
simplifying the *MetadataQuorumCommand* by leveraging the Admin API for
adding and removing controllers.
You can find the details here: https://cwiki.apache.org/confluence/x/PZFEF.
I believe this proposal could provide a better user experience,
and I’d really appreciate it if you could take a look and share your
thoughts or feedback.
Best regards,
Kuan-Po Tseng
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by José Armando García Sancio <js...@confluent.io.INVALID>.
Hi Kuan,
Thanks for your suggestion to improve the UX for adding and removing
controllers. I skimmed the KIP and I have some feedback. I'll spend
some time tomorrow to provide you with detailed feedback.
Thanks!
--
-José
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Kuan Po Tseng <br...@apache.org>.
Hi all,
If there are no further opinions, I would like to start the vote tomorrow. Thank you!
Best,
Kuan-Po Tseng
On 2025/03/31 00:54:12 Kuan Po Tseng wrote:
> Hi Chia-Ping,
>
> Thank you for your feedback! I agree with you that deprecating those options in 4.x would be a bit too soon. I’ve added these points to the Alternatives section. Thanks again!
>
> Best,
> Kuan-Po Tseng
>
> On 2025/03/30 19:22:12 Chia-Ping Tsai wrote:
> > hi Kuan-Po
> >
> > > I've been considering it, but I'm unsure if it's the right move. I'd appreciate hearing more opinions on this.
> >
> > I prefer to keep the way of using properties, as it makes sense if users already have controller properties in the local. Also, the command line is released with 4.0, so it is weird that we deprecate it in 4.x. Maybe we can add this discussion to the "Rejected Alternatives"
> >
> > Best,
> > Chia-Ping
> >
> >
> > On 2025/03/30 10:37:41 Kuan Po Tseng wrote:
> > > Hi Federico,
> > >
> > > > - the Jira link needs to be fixed
> > > Updated, thanks.
> > >
> > > > - usually discussion threads starts with [DISCUSS] prefix
> > > My apologies, I'll make sure to use it next time.
> > >
> > > > - have you considered deprecating the old way of adding/removing controllers?
> > > I've been considering it, but I'm unsure if it's the right move. I'd appreciate hearing more opinions on this.
> > >
> > > On 2025/03/25 08:49:07 Federico Valeri wrote:
> > > > Hi, thanks for the KIP. This is a nice improvement IMO.
> > > >
> > > > Some minor notes:
> > > > - the Jira link needs to be fixed
> > > > - usually discussion threads starts with [DISCUSS] prefix
> > > > - have you considered deprecating the old way of adding/removing controllers?
> > > >
> > > >
> > > > On Mon, Mar 10, 2025 at 2:27 AM Kuan Po Tseng <br...@gmail.com> wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to start a discussion thread on *KIP-1141*, which proposes a
> > > > > relatively small but useful change:
> > > > >
> > > > > simplifying the *MetadataQuorumCommand* by leveraging the Admin API for
> > > > > adding and removing controllers.
> > > > >
> > > > > You can find the details here: https://cwiki.apache.org/confluence/x/PZFEF.
> > > > >
> > > > > I believe this proposal could provide a better user experience,
> > > > >
> > > > > and I’d really appreciate it if you could take a look and share your
> > > > > thoughts or feedback.
> > > > >
> > > > > Best regards,
> > > > > Kuan-Po Tseng
> > > >
> > >
> >
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Kuan Po Tseng <br...@apache.org>.
Hi Chia-Ping,
Thank you for your feedback! I agree with you that deprecating those options in 4.x would be a bit too soon. I’ve added these points to the Alternatives section. Thanks again!
Best,
Kuan-Po Tseng
On 2025/03/30 19:22:12 Chia-Ping Tsai wrote:
> hi Kuan-Po
>
> > I've been considering it, but I'm unsure if it's the right move. I'd appreciate hearing more opinions on this.
>
> I prefer to keep the way of using properties, as it makes sense if users already have controller properties in the local. Also, the command line is released with 4.0, so it is weird that we deprecate it in 4.x. Maybe we can add this discussion to the "Rejected Alternatives"
>
> Best,
> Chia-Ping
>
>
> On 2025/03/30 10:37:41 Kuan Po Tseng wrote:
> > Hi Federico,
> >
> > > - the Jira link needs to be fixed
> > Updated, thanks.
> >
> > > - usually discussion threads starts with [DISCUSS] prefix
> > My apologies, I'll make sure to use it next time.
> >
> > > - have you considered deprecating the old way of adding/removing controllers?
> > I've been considering it, but I'm unsure if it's the right move. I'd appreciate hearing more opinions on this.
> >
> > On 2025/03/25 08:49:07 Federico Valeri wrote:
> > > Hi, thanks for the KIP. This is a nice improvement IMO.
> > >
> > > Some minor notes:
> > > - the Jira link needs to be fixed
> > > - usually discussion threads starts with [DISCUSS] prefix
> > > - have you considered deprecating the old way of adding/removing controllers?
> > >
> > >
> > > On Mon, Mar 10, 2025 at 2:27 AM Kuan Po Tseng <br...@gmail.com> wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I'd like to start a discussion thread on *KIP-1141*, which proposes a
> > > > relatively small but useful change:
> > > >
> > > > simplifying the *MetadataQuorumCommand* by leveraging the Admin API for
> > > > adding and removing controllers.
> > > >
> > > > You can find the details here: https://cwiki.apache.org/confluence/x/PZFEF.
> > > >
> > > > I believe this proposal could provide a better user experience,
> > > >
> > > > and I’d really appreciate it if you could take a look and share your
> > > > thoughts or feedback.
> > > >
> > > > Best regards,
> > > > Kuan-Po Tseng
> > >
> >
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Kuan-Po
> I've been considering it, but I'm unsure if it's the right move. I'd appreciate hearing more opinions on this.
I prefer to keep the way of using properties, as it makes sense if users already have controller properties in the local. Also, the command line is released with 4.0, so it is weird that we deprecate it in 4.x. Maybe we can add this discussion to the "Rejected Alternatives"
Best,
Chia-Ping
On 2025/03/30 10:37:41 Kuan Po Tseng wrote:
> Hi Federico,
>
> > - the Jira link needs to be fixed
> Updated, thanks.
>
> > - usually discussion threads starts with [DISCUSS] prefix
> My apologies, I'll make sure to use it next time.
>
> > - have you considered deprecating the old way of adding/removing controllers?
> I've been considering it, but I'm unsure if it's the right move. I'd appreciate hearing more opinions on this.
>
> On 2025/03/25 08:49:07 Federico Valeri wrote:
> > Hi, thanks for the KIP. This is a nice improvement IMO.
> >
> > Some minor notes:
> > - the Jira link needs to be fixed
> > - usually discussion threads starts with [DISCUSS] prefix
> > - have you considered deprecating the old way of adding/removing controllers?
> >
> >
> > On Mon, Mar 10, 2025 at 2:27 AM Kuan Po Tseng <br...@gmail.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > I'd like to start a discussion thread on *KIP-1141*, which proposes a
> > > relatively small but useful change:
> > >
> > > simplifying the *MetadataQuorumCommand* by leveraging the Admin API for
> > > adding and removing controllers.
> > >
> > > You can find the details here: https://cwiki.apache.org/confluence/x/PZFEF.
> > >
> > > I believe this proposal could provide a better user experience,
> > >
> > > and I’d really appreciate it if you could take a look and share your
> > > thoughts or feedback.
> > >
> > > Best regards,
> > > Kuan-Po Tseng
> >
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Kuan Po Tseng <br...@apache.org>.
Hi Federico,
> - the Jira link needs to be fixed
Updated, thanks.
> - usually discussion threads starts with [DISCUSS] prefix
My apologies, I'll make sure to use it next time.
> - have you considered deprecating the old way of adding/removing controllers?
I've been considering it, but I'm unsure if it's the right move. I'd appreciate hearing more opinions on this.
On 2025/03/25 08:49:07 Federico Valeri wrote:
> Hi, thanks for the KIP. This is a nice improvement IMO.
>
> Some minor notes:
> - the Jira link needs to be fixed
> - usually discussion threads starts with [DISCUSS] prefix
> - have you considered deprecating the old way of adding/removing controllers?
>
>
> On Mon, Mar 10, 2025 at 2:27 AM Kuan Po Tseng <br...@gmail.com> wrote:
> >
> > Hi everyone,
> >
> > I'd like to start a discussion thread on *KIP-1141*, which proposes a
> > relatively small but useful change:
> >
> > simplifying the *MetadataQuorumCommand* by leveraging the Admin API for
> > adding and removing controllers.
> >
> > You can find the details here: https://cwiki.apache.org/confluence/x/PZFEF.
> >
> > I believe this proposal could provide a better user experience,
> >
> > and I’d really appreciate it if you could take a look and share your
> > thoughts or feedback.
> >
> > Best regards,
> > Kuan-Po Tseng
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Federico Valeri <fe...@gmail.com>.
Hi, thanks for the KIP. This is a nice improvement IMO.
Some minor notes:
- the Jira link needs to be fixed
- usually discussion threads starts with [DISCUSS] prefix
- have you considered deprecating the old way of adding/removing controllers?
On Mon, Mar 10, 2025 at 2:27 AM Kuan Po Tseng <br...@gmail.com> wrote:
>
> Hi everyone,
>
> I'd like to start a discussion thread on *KIP-1141*, which proposes a
> relatively small but useful change:
>
> simplifying the *MetadataQuorumCommand* by leveraging the Admin API for
> adding and removing controllers.
>
> You can find the details here: https://cwiki.apache.org/confluence/x/PZFEF.
>
> I believe this proposal could provide a better user experience,
>
> and I’d really appreciate it if you could take a look and share your
> thoughts or feedback.
>
> Best regards,
> Kuan-Po Tseng
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Kuan-Po Tseng <br...@gmail.com>.
Hi all,
I updated KIP-1141 based on a good suggestion from Chia-Ping in the PR
review:
https://github.com/apache/kafka/pull/22531#discussion_r3488220386
The idea is that, with this KIP, the directory UUID and endpoints become
optional because the controller can derive them from its in-memory state.
Therefore, it is more natural to move these fields into the options objects:
addRaftVoter(int voterId, AddRaftVoterOptions options)
removeRaftVoter(int voterId, RemoveRaftVoterOptions options)
AddRaftVoterOptions will include optional voterDirectoryId and endpoints;
RemoveRaftVoterOptions will include optional voterDirectoryId.
The existing positional overloads will remain for compatibility, but will
be deprecated and planned for removal in Apache Kafka 5.0.
Please check the latest KIP and let me know if you have any concerns. Thank
you!
Best,
Kuan Po
On Tue, Aug 12, 2025 at 9:27 PM Kuan-Po Tseng <br...@gmail.com> wrote:
> Hi Chia-Ping,
>
> I still think this KIP could be useful. Say a user doesn’t want to turn on
> controller.quorum.auto.join.enable.
> One example might be if they have pre-staging controllers running as
> observers (like DR or standby)
> and don’t want them voting yet. With auto-join on, any new controller
> would just promote itself to a voter
> instead of staying an observer.
>
> In that case, you’d still have to manually promote the observer when
> you’re ready.
> To me, that sounds like a valid scenario — but maybe I’m overthinking it
> and such cases don’t really happen.
> If that’s true, then sure, we can skip the extra complexity and drop this
> KIP.
>
> Best,
> Kuan-Po Tseng
>
> On Tue, Aug 12, 2025 at 2:27 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
>
>> hi Kuan-Po
>>
>> I'm thinking about the value of this KIP after
>> controller.quorum.auto.join.enable is enabled.
>> WDYT?
>>
>> Best,
>> Chia-Ping
>>
>> Kuan Po Tseng <br...@apache.org> 於 2025年8月10日 週日 下午3:36寫道：
>>
>> > Hi everyone,
>> >
>> > I've updated the KIP with the following changes:
>> >
>> > - Using the in-memory LeaderState and ClusterImage to
>> > retrieve the directory UUID and controller endpoints. The previous
>> > approach has been moved to the "Rejected Alternatives" section.
>> >
>> > - When both --command-config and --controller-id are provided in
>> > the add-controller command, the config file will only be applied
>> > during Admin initialization.
>> >
>> > One point for discussion:
>> > When adding controllers, there could be a scenario where observers
>> > have different directory UUIDs but share the same node ID. In such
>> > cases, I propose returning a non-retryable IllegalStateException and
>> > notifying the user that they should explicitly specify the directory
>> > UUID when adding a voter.
>> >
>> > In KIP-853, this scenario can occur during a node disk failure and is
>> > also permitted during Kafka controller startup. For example, we can
>> > start a standalone controller with node ID 3000, then start two
>> > observers with the same node ID 3001 but different directory UUIDs.
>> >
>> > Best,
>> > Kuan-Po Tseng
>> >
>> > On 2025/08/01 07:47:43 Luke Chen wrote:
>> > > Hi Kuan-Po,
>> > >
>> > > > L03
>> > > > In KAFKA-19563, you mentioned introducing a new
>> > > --add-controller-command-config option to separate admin and
>> > > controller configs. That sounds like a solid approach.
>> > >
>> > > Yes, that can resolve the issue in KAFKA-19563, but it conflicts with
>> the
>> > > motivation in the KIP:
>> > > 1. Limited Accessibility: The node executing the tool must have
>> direct
>> > > access to the metadata path of the node being added or removed. This
>> > > restricts the ability to use node A to manage node B, as node A may
>> not
>> > > have access to the metadata folder on node B.
>> > > 2. Dependency on Node Configuration: The tool requires access to
>> the
>> > > configuration of the node being managed.
>> > >
>> > > > That said, I was thinking of an alternative:
>> > > what if we change the logic so that *when both* --command-config
>> > > *and* --controller-id are provided, the command config only applies
>> > > to admin initialization? Users can add extra admin properties in
>> > > --command-config.
>> > >
>> > > I'm good with it.
>> > >
>> > > >L04
>> > > > As jose mentioned, the controller (observer) endpoints are stored in
>> > the
>> > > ClusterImage
>> > >
>> > > Yes, I like this idea since we already have endpoint information
>> stored
>> > in
>> > > the active controller.
>> > >
>> > > Thank you.
>> > > Luke
>> > >
>> > >
>> > > On Fri, Aug 1, 2025 at 12:40 AM Chia-Ping Tsai <ch...@gmail.com>
>> > wrote:
>> > >
>> > > > hi Kuan-Po
>> > > >
>> > > > Yeah, I agree — to make things easier, we could enhance the logic.
>> > > > If the user provides --bootstrap-server, we can retrieve
>> > > > controller.quorum.bootstrap.servers from the broker configs
>> > > > using admin.describeConfigs, and use that to get the required
>> > > > controller configs.
>> > > >
>> > > >
>> > > > As jose mentioned, the controller (observer) endpoints are stored in
>> > the
>> > > > ClusterImage.
>> > > >
>> > > > Another (more complicated) approach would be to make the endpoints
>> in
>> > > > AddRaftVoterRequest optional.
>> > > >
>> > > > Initially, I thought this KIP could address this issue in a simple
>> way,
>> > > > without requiring any changes to the RPC :)
>> > > >
>> > > > Best,
>> > > > Chia-Ping
>> > > >
>> > > > Kuan-Po Tseng <br...@gmail.com> 於 2025年8月1日 週五 上午12:05寫道：
>> > > >
>> > > > > Hi Luke,
>> > > > >
>> > > > > Appreciate the feedback — glad to finally see some voices in this
>> > thread!
>> > > > > :)
>> > > > >
>> > > > > *L01, L02:*
>> > > > > Sure thing!
>> > > > >
>> > > > > *L03:*
>> > > > > Thanks for the heads-up — I hadn’t thought of that before.
>> > > > > In KAFKA-19563, you mentioned introducing a new
>> > > > > --add-controller-command-config option to separate admin and
>> > > > > controller configs. That sounds like a solid approach.
>> > > > >
>> > > > > That said, I was thinking of an alternative:
>> > > > > what if we change the logic so that *when both* --command-config
>> > > > > *and* --controller-id are provided, the command config only
>> applies
>> > > > > to admin initialization? Users can add extra admin properties in
>> > > > > --command-config.
>> > > > >
>> > > > > What do you think?
>> > > > >
>> > > > > *L04:*
>> > > > > Yeah, I agree — to make things easier, we could enhance the logic.
>> > > > > If the user provides --bootstrap-server, we can retrieve
>> > > > > controller.quorum.bootstrap.servers from the broker configs
>> > > > > using admin.describeConfigs, and use that to get the required
>> > > > > controller configs.
>> > > > >
>> > > > > Then we could allow users to run something like:
>> > > > >
>> > > > > bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 \
>> > > > > add-controller --controller-id <id>
>> > > > >
>> > > > >
>> > > > > Best,
>> > > > > Kuan-Po
>> > > > >
>> > > > > On Thu, Jul 31, 2025 at 3:24 PM Luke Chen <sh...@gmail.com>
>> wrote:
>> > > > >
>> > > > > > Hi Kuan-Po,
>> > > > > >
>> > > > > > Thanks for the KIP!
>> > > > > > We also faced the similar issue recently.
>> > > > > >
>> > > > > > Some suggestions/questions:
>> > > > > > 1. Could you also include the issue in KAFKA-19563
>> > > > > > <https://issues.apache.org/jira/browse/KAFKA-19563> as the
>> > motivation
>> > > > in
>> > > > > > the KIP?
>> > > > > > 2. Could you make it clear in the KIP that after the KIP, the
>> > > > > > add-controller request from kafka-metadata-quorum.sh will take
>> more
>> > > > time
>> > > > > > than before because of more API calls needed?
>> > > > > > 3. "If —-command-config is provided, fallback to the existing
>> > behavior,
>> > > > > and
>> > > > > > —-controller-id option will be ignored."
>> > > > > > On this, it looks like it still doesn't resolve the issue
>> > described in
>> > > > > > KAFKA-19563 <https://issues.apache.org/jira/browse/KAFKA-19563>
>> > > > because
>> > > > > if
>> > > > > > the client wants to pass the "client.id", and provide a
>> properties
>> > > > file
>> > > > > in
>> > > > > > --command-config, it 'll be treated as the controller.properties
>> > and
>> > > > fail
>> > > > > > the request, right? Ex:
>> > > > > > `bin/kafka-metadata-quorum.sh --bootstrap-controller
>> localhost:9093
>> > > > > > add-controller --controller-id <id> --command-config
>> > admin.properties`
>> > > > > > 4. "This can only be done with bootstrap controller option
>> since we
>> > > > can‘t
>> > > > > > use bootstrap.server in Admin#describeConfigs to get controller
>> > > > configs."
>> > > > > > I understand the reason but it makes the script more confusing
>> in
>> > my
>> > > > > > opinion.
>> > > > > > I'm wondering why we need the controller endpoint when talking
>> > with the
>> > > > > > broker with --bootstrap-server? The controller endpoint should
>> > already
>> > > > > > exist in broker itself, right?
>> > > > > >
>> > > > > > Thank you.
>> > > > > > Luke
>> > > > > >
>> > > > > > On Sat, Apr 19, 2025 at 11:32 PM Kuan Po Tseng <
>> > brandboat@apache.org>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi José,
>> > > > > > >
>> > > > > > > Apologies for the delayed response. Do you still have any
>> > questions
>> > > > > > > regarding the describeConfig part? As Chia-Ping mentioned, we
>> are
>> > > > able
>> > > > > to
>> > > > > > > retrieve all broker configurations through that method. (
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> >
>> https://github.com/apache/kafka/blob/da46cf6e79afbbed1da2bae831e0f70992e85f9b/core/src/main/scala/kafka/server/ConfigHelper.scala#L121-L123
>> > > > > > > )
>> > > > > > >
>> > > > > > > Please feel free to reach out if you have any further
>> questions
>> > or
>> > > > need
>> > > > > > > clarification. Thank you again for your valuable feedback!
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Kuan-Po Tseng
>> > > > > > >
>> > > > > > > On 2025/04/03 16:04:36 José Armando García Sancio wrote:
>> > > > > > > > Hi Chia,
>> > > > > > > >
>> > > > > > > > On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <
>> > > > chia7712@apache.org>
>> > > > > > > wrote:
>> > > > > > > > > We propose to use `Admin#describeConfigs` to get the
>> configs
>> > for
>> > > > > > > specific controller if the bootstrap.controllers is
>> configured.
>> > This
>> > > > > > > approach is similar to what `MetadataQuorumCommand` does, and
>> the
>> > > > > > > difference is `MetadataQuorumCommand` read those configs from
>> > local
>> > > > > file
>> > > > > > > and this KIP gets those configs by `Admin#describeConfigs`
>> > > > > > > >
>> > > > > > > > I am not sure. I have to look at that code but doesn't
>> > > > > > > > "Admin#describeConfigs" only return dynamic configuration
>> for
>> > the
>> > > > > > > > controller? Most users configure the controller using the
>> > server
>> > > > > > > > properties file. My current understanding is that values
>> coming
>> > > > from
>> > > > > > > > the properties file won't show up in Admin#describeConfigs.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > --
>> > > > > > > > -José
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Kuan-Po Tseng <br...@gmail.com>.
Hi Chia-Ping,
I still think this KIP could be useful. Say a user doesn’t want to turn on
controller.quorum.auto.join.enable.
One example might be if they have pre-staging controllers running as
observers (like DR or standby)
and don’t want them voting yet. With auto-join on, any new controller would
just promote itself to a voter
instead of staying an observer.
In that case, you’d still have to manually promote the observer when you’re
ready.
To me, that sounds like a valid scenario — but maybe I’m overthinking it
and such cases don’t really happen.
If that’s true, then sure, we can skip the extra complexity and drop this
KIP.
Best,
Kuan-Po Tseng
On Tue, Aug 12, 2025 at 2:27 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
> hi Kuan-Po
>
> I'm thinking about the value of this KIP after
> controller.quorum.auto.join.enable is enabled.
> WDYT?
>
> Best,
> Chia-Ping
>
> Kuan Po Tseng <br...@apache.org> 於 2025年8月10日 週日 下午3:36寫道：
>
> > Hi everyone,
> >
> > I've updated the KIP with the following changes:
> >
> > - Using the in-memory LeaderState and ClusterImage to
> > retrieve the directory UUID and controller endpoints. The previous
> > approach has been moved to the "Rejected Alternatives" section.
> >
> > - When both --command-config and --controller-id are provided in
> > the add-controller command, the config file will only be applied
> > during Admin initialization.
> >
> > One point for discussion:
> > When adding controllers, there could be a scenario where observers
> > have different directory UUIDs but share the same node ID. In such
> > cases, I propose returning a non-retryable IllegalStateException and
> > notifying the user that they should explicitly specify the directory
> > UUID when adding a voter.
> >
> > In KIP-853, this scenario can occur during a node disk failure and is
> > also permitted during Kafka controller startup. For example, we can
> > start a standalone controller with node ID 3000, then start two
> > observers with the same node ID 3001 but different directory UUIDs.
> >
> > Best,
> > Kuan-Po Tseng
> >
> > On 2025/08/01 07:47:43 Luke Chen wrote:
> > > Hi Kuan-Po,
> > >
> > > > L03
> > > > In KAFKA-19563, you mentioned introducing a new
> > > --add-controller-command-config option to separate admin and
> > > controller configs. That sounds like a solid approach.
> > >
> > > Yes, that can resolve the issue in KAFKA-19563, but it conflicts with
> the
> > > motivation in the KIP:
> > > 1. Limited Accessibility: The node executing the tool must have
> direct
> > > access to the metadata path of the node being added or removed. This
> > > restricts the ability to use node A to manage node B, as node A may not
> > > have access to the metadata folder on node B.
> > > 2. Dependency on Node Configuration: The tool requires access to the
> > > configuration of the node being managed.
> > >
> > > > That said, I was thinking of an alternative:
> > > what if we change the logic so that *when both* --command-config
> > > *and* --controller-id are provided, the command config only applies
> > > to admin initialization? Users can add extra admin properties in
> > > --command-config.
> > >
> > > I'm good with it.
> > >
> > > >L04
> > > > As jose mentioned, the controller (observer) endpoints are stored in
> > the
> > > ClusterImage
> > >
> > > Yes, I like this idea since we already have endpoint information stored
> > in
> > > the active controller.
> > >
> > > Thank you.
> > > Luke
> > >
> > >
> > > On Fri, Aug 1, 2025 at 12:40 AM Chia-Ping Tsai <ch...@gmail.com>
> > wrote:
> > >
> > > > hi Kuan-Po
> > > >
> > > > Yeah, I agree — to make things easier, we could enhance the logic.
> > > > If the user provides --bootstrap-server, we can retrieve
> > > > controller.quorum.bootstrap.servers from the broker configs
> > > > using admin.describeConfigs, and use that to get the required
> > > > controller configs.
> > > >
> > > >
> > > > As jose mentioned, the controller (observer) endpoints are stored in
> > the
> > > > ClusterImage.
> > > >
> > > > Another (more complicated) approach would be to make the endpoints in
> > > > AddRaftVoterRequest optional.
> > > >
> > > > Initially, I thought this KIP could address this issue in a simple
> way,
> > > > without requiring any changes to the RPC :)
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > Kuan-Po Tseng <br...@gmail.com> 於 2025年8月1日 週五 上午12:05寫道：
> > > >
> > > > > Hi Luke,
> > > > >
> > > > > Appreciate the feedback — glad to finally see some voices in this
> > thread!
> > > > > :)
> > > > >
> > > > > *L01, L02:*
> > > > > Sure thing!
> > > > >
> > > > > *L03:*
> > > > > Thanks for the heads-up — I hadn’t thought of that before.
> > > > > In KAFKA-19563, you mentioned introducing a new
> > > > > --add-controller-command-config option to separate admin and
> > > > > controller configs. That sounds like a solid approach.
> > > > >
> > > > > That said, I was thinking of an alternative:
> > > > > what if we change the logic so that *when both* --command-config
> > > > > *and* --controller-id are provided, the command config only applies
> > > > > to admin initialization? Users can add extra admin properties in
> > > > > --command-config.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > *L04:*
> > > > > Yeah, I agree — to make things easier, we could enhance the logic.
> > > > > If the user provides --bootstrap-server, we can retrieve
> > > > > controller.quorum.bootstrap.servers from the broker configs
> > > > > using admin.describeConfigs, and use that to get the required
> > > > > controller configs.
> > > > >
> > > > > Then we could allow users to run something like:
> > > > >
> > > > > bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 \
> > > > > add-controller --controller-id <id>
> > > > >
> > > > >
> > > > > Best,
> > > > > Kuan-Po
> > > > >
> > > > > On Thu, Jul 31, 2025 at 3:24 PM Luke Chen <sh...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Kuan-Po,
> > > > > >
> > > > > > Thanks for the KIP!
> > > > > > We also faced the similar issue recently.
> > > > > >
> > > > > > Some suggestions/questions:
> > > > > > 1. Could you also include the issue in KAFKA-19563
> > > > > > <https://issues.apache.org/jira/browse/KAFKA-19563> as the
> > motivation
> > > > in
> > > > > > the KIP?
> > > > > > 2. Could you make it clear in the KIP that after the KIP, the
> > > > > > add-controller request from kafka-metadata-quorum.sh will take
> more
> > > > time
> > > > > > than before because of more API calls needed?
> > > > > > 3. "If —-command-config is provided, fallback to the existing
> > behavior,
> > > > > and
> > > > > > —-controller-id option will be ignored."
> > > > > > On this, it looks like it still doesn't resolve the issue
> > described in
> > > > > > KAFKA-19563 <https://issues.apache.org/jira/browse/KAFKA-19563>
> > > > because
> > > > > if
> > > > > > the client wants to pass the "client.id", and provide a
> properties
> > > > file
> > > > > in
> > > > > > --command-config, it 'll be treated as the controller.properties
> > and
> > > > fail
> > > > > > the request, right? Ex:
> > > > > > `bin/kafka-metadata-quorum.sh --bootstrap-controller
> localhost:9093
> > > > > > add-controller --controller-id <id> --command-config
> > admin.properties`
> > > > > > 4. "This can only be done with bootstrap controller option since
> we
> > > > can‘t
> > > > > > use bootstrap.server in Admin#describeConfigs to get controller
> > > > configs."
> > > > > > I understand the reason but it makes the script more confusing in
> > my
> > > > > > opinion.
> > > > > > I'm wondering why we need the controller endpoint when talking
> > with the
> > > > > > broker with --bootstrap-server? The controller endpoint should
> > already
> > > > > > exist in broker itself, right?
> > > > > >
> > > > > > Thank you.
> > > > > > Luke
> > > > > >
> > > > > > On Sat, Apr 19, 2025 at 11:32 PM Kuan Po Tseng <
> > brandboat@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi José,
> > > > > > >
> > > > > > > Apologies for the delayed response. Do you still have any
> > questions
> > > > > > > regarding the describeConfig part? As Chia-Ping mentioned, we
> are
> > > > able
> > > > > to
> > > > > > > retrieve all broker configurations through that method. (
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/apache/kafka/blob/da46cf6e79afbbed1da2bae831e0f70992e85f9b/core/src/main/scala/kafka/server/ConfigHelper.scala#L121-L123
> > > > > > > )
> > > > > > >
> > > > > > > Please feel free to reach out if you have any further questions
> > or
> > > > need
> > > > > > > clarification. Thank you again for your valuable feedback!
> > > > > > >
> > > > > > > Best,
> > > > > > > Kuan-Po Tseng
> > > > > > >
> > > > > > > On 2025/04/03 16:04:36 José Armando García Sancio wrote:
> > > > > > > > Hi Chia,
> > > > > > > >
> > > > > > > > On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <
> > > > chia7712@apache.org>
> > > > > > > wrote:
> > > > > > > > > We propose to use `Admin#describeConfigs` to get the
> configs
> > for
> > > > > > > specific controller if the bootstrap.controllers is configured.
> > This
> > > > > > > approach is similar to what `MetadataQuorumCommand` does, and
> the
> > > > > > > difference is `MetadataQuorumCommand` read those configs from
> > local
> > > > > file
> > > > > > > and this KIP gets those configs by `Admin#describeConfigs`
> > > > > > > >
> > > > > > > > I am not sure. I have to look at that code but doesn't
> > > > > > > > "Admin#describeConfigs" only return dynamic configuration for
> > the
> > > > > > > > controller? Most users configure the controller using the
> > server
> > > > > > > > properties file. My current understanding is that values
> coming
> > > > from
> > > > > > > > the properties file won't show up in Admin#describeConfigs.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > --
> > > > > > > > -José
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi Kuan-Po
I'm thinking about the value of this KIP after
controller.quorum.auto.join.enable is enabled.
WDYT?
Best,
Chia-Ping
Kuan Po Tseng <br...@apache.org> 於 2025年8月10日 週日 下午3:36寫道：
> Hi everyone,
>
> I've updated the KIP with the following changes:
>
> - Using the in-memory LeaderState and ClusterImage to
> retrieve the directory UUID and controller endpoints. The previous
> approach has been moved to the "Rejected Alternatives" section.
>
> - When both --command-config and --controller-id are provided in
> the add-controller command, the config file will only be applied
> during Admin initialization.
>
> One point for discussion:
> When adding controllers, there could be a scenario where observers
> have different directory UUIDs but share the same node ID. In such
> cases, I propose returning a non-retryable IllegalStateException and
> notifying the user that they should explicitly specify the directory
> UUID when adding a voter.
>
> In KIP-853, this scenario can occur during a node disk failure and is
> also permitted during Kafka controller startup. For example, we can
> start a standalone controller with node ID 3000, then start two
> observers with the same node ID 3001 but different directory UUIDs.
>
> Best,
> Kuan-Po Tseng
>
> On 2025/08/01 07:47:43 Luke Chen wrote:
> > Hi Kuan-Po,
> >
> > > L03
> > > In KAFKA-19563, you mentioned introducing a new
> > --add-controller-command-config option to separate admin and
> > controller configs. That sounds like a solid approach.
> >
> > Yes, that can resolve the issue in KAFKA-19563, but it conflicts with the
> > motivation in the KIP:
> > 1. Limited Accessibility: The node executing the tool must have direct
> > access to the metadata path of the node being added or removed. This
> > restricts the ability to use node A to manage node B, as node A may not
> > have access to the metadata folder on node B.
> > 2. Dependency on Node Configuration: The tool requires access to the
> > configuration of the node being managed.
> >
> > > That said, I was thinking of an alternative:
> > what if we change the logic so that *when both* --command-config
> > *and* --controller-id are provided, the command config only applies
> > to admin initialization? Users can add extra admin properties in
> > --command-config.
> >
> > I'm good with it.
> >
> > >L04
> > > As jose mentioned, the controller (observer) endpoints are stored in
> the
> > ClusterImage
> >
> > Yes, I like this idea since we already have endpoint information stored
> in
> > the active controller.
> >
> > Thank you.
> > Luke
> >
> >
> > On Fri, Aug 1, 2025 at 12:40 AM Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >
> > > hi Kuan-Po
> > >
> > > Yeah, I agree — to make things easier, we could enhance the logic.
> > > If the user provides --bootstrap-server, we can retrieve
> > > controller.quorum.bootstrap.servers from the broker configs
> > > using admin.describeConfigs, and use that to get the required
> > > controller configs.
> > >
> > >
> > > As jose mentioned, the controller (observer) endpoints are stored in
> the
> > > ClusterImage.
> > >
> > > Another (more complicated) approach would be to make the endpoints in
> > > AddRaftVoterRequest optional.
> > >
> > > Initially, I thought this KIP could address this issue in a simple way,
> > > without requiring any changes to the RPC :)
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > Kuan-Po Tseng <br...@gmail.com> 於 2025年8月1日 週五 上午12:05寫道：
> > >
> > > > Hi Luke,
> > > >
> > > > Appreciate the feedback — glad to finally see some voices in this
> thread!
> > > > :)
> > > >
> > > > *L01, L02:*
> > > > Sure thing!
> > > >
> > > > *L03:*
> > > > Thanks for the heads-up — I hadn’t thought of that before.
> > > > In KAFKA-19563, you mentioned introducing a new
> > > > --add-controller-command-config option to separate admin and
> > > > controller configs. That sounds like a solid approach.
> > > >
> > > > That said, I was thinking of an alternative:
> > > > what if we change the logic so that *when both* --command-config
> > > > *and* --controller-id are provided, the command config only applies
> > > > to admin initialization? Users can add extra admin properties in
> > > > --command-config.
> > > >
> > > > What do you think?
> > > >
> > > > *L04:*
> > > > Yeah, I agree — to make things easier, we could enhance the logic.
> > > > If the user provides --bootstrap-server, we can retrieve
> > > > controller.quorum.bootstrap.servers from the broker configs
> > > > using admin.describeConfigs, and use that to get the required
> > > > controller configs.
> > > >
> > > > Then we could allow users to run something like:
> > > >
> > > > bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 \
> > > > add-controller --controller-id <id>
> > > >
> > > >
> > > > Best,
> > > > Kuan-Po
> > > >
> > > > On Thu, Jul 31, 2025 at 3:24 PM Luke Chen <sh...@gmail.com> wrote:
> > > >
> > > > > Hi Kuan-Po,
> > > > >
> > > > > Thanks for the KIP!
> > > > > We also faced the similar issue recently.
> > > > >
> > > > > Some suggestions/questions:
> > > > > 1. Could you also include the issue in KAFKA-19563
> > > > > <https://issues.apache.org/jira/browse/KAFKA-19563> as the
> motivation
> > > in
> > > > > the KIP?
> > > > > 2. Could you make it clear in the KIP that after the KIP, the
> > > > > add-controller request from kafka-metadata-quorum.sh will take more
> > > time
> > > > > than before because of more API calls needed?
> > > > > 3. "If —-command-config is provided, fallback to the existing
> behavior,
> > > > and
> > > > > —-controller-id option will be ignored."
> > > > > On this, it looks like it still doesn't resolve the issue
> described in
> > > > > KAFKA-19563 <https://issues.apache.org/jira/browse/KAFKA-19563>
> > > because
> > > > if
> > > > > the client wants to pass the "client.id", and provide a properties
> > > file
> > > > in
> > > > > --command-config, it 'll be treated as the controller.properties
> and
> > > fail
> > > > > the request, right? Ex:
> > > > > `bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093
> > > > > add-controller --controller-id <id> --command-config
> admin.properties`
> > > > > 4. "This can only be done with bootstrap controller option since we
> > > can‘t
> > > > > use bootstrap.server in Admin#describeConfigs to get controller
> > > configs."
> > > > > I understand the reason but it makes the script more confusing in
> my
> > > > > opinion.
> > > > > I'm wondering why we need the controller endpoint when talking
> with the
> > > > > broker with --bootstrap-server? The controller endpoint should
> already
> > > > > exist in broker itself, right?
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Sat, Apr 19, 2025 at 11:32 PM Kuan Po Tseng <
> brandboat@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi José,
> > > > > >
> > > > > > Apologies for the delayed response. Do you still have any
> questions
> > > > > > regarding the describeConfig part? As Chia-Ping mentioned, we are
> > > able
> > > > to
> > > > > > retrieve all broker configurations through that method. (
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/kafka/blob/da46cf6e79afbbed1da2bae831e0f70992e85f9b/core/src/main/scala/kafka/server/ConfigHelper.scala#L121-L123
> > > > > > )
> > > > > >
> > > > > > Please feel free to reach out if you have any further questions
> or
> > > need
> > > > > > clarification. Thank you again for your valuable feedback!
> > > > > >
> > > > > > Best,
> > > > > > Kuan-Po Tseng
> > > > > >
> > > > > > On 2025/04/03 16:04:36 José Armando García Sancio wrote:
> > > > > > > Hi Chia,
> > > > > > >
> > > > > > > On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <
> > > chia7712@apache.org>
> > > > > > wrote:
> > > > > > > > We propose to use `Admin#describeConfigs` to get the configs
> for
> > > > > > specific controller if the bootstrap.controllers is configured.
> This
> > > > > > approach is similar to what `MetadataQuorumCommand` does, and the
> > > > > > difference is `MetadataQuorumCommand` read those configs from
> local
> > > > file
> > > > > > and this KIP gets those configs by `Admin#describeConfigs`
> > > > > > >
> > > > > > > I am not sure. I have to look at that code but doesn't
> > > > > > > "Admin#describeConfigs" only return dynamic configuration for
> the
> > > > > > > controller? Most users configure the controller using the
> server
> > > > > > > properties file. My current understanding is that values coming
> > > from
> > > > > > > the properties file won't show up in Admin#describeConfigs.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > --
> > > > > > > -José
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Kuan Po Tseng <br...@apache.org>.
Hi everyone,
I've updated the KIP with the following changes:
- Using the in-memory LeaderState and ClusterImage to
retrieve the directory UUID and controller endpoints. The previous
approach has been moved to the "Rejected Alternatives" section.
- When both --command-config and --controller-id are provided in
the add-controller command, the config file will only be applied
during Admin initialization.
One point for discussion:
When adding controllers, there could be a scenario where observers
have different directory UUIDs but share the same node ID. In such
cases, I propose returning a non-retryable IllegalStateException and
notifying the user that they should explicitly specify the directory
UUID when adding a voter.
In KIP-853, this scenario can occur during a node disk failure and is
also permitted during Kafka controller startup. For example, we can
start a standalone controller with node ID 3000, then start two
observers with the same node ID 3001 but different directory UUIDs.
Best,
Kuan-Po Tseng
On 2025/08/01 07:47:43 Luke Chen wrote:
> Hi Kuan-Po,
>
> > L03
> > In KAFKA-19563, you mentioned introducing a new
> --add-controller-command-config option to separate admin and
> controller configs. That sounds like a solid approach.
>
> Yes, that can resolve the issue in KAFKA-19563, but it conflicts with the
> motivation in the KIP:
> 1. Limited Accessibility: The node executing the tool must have direct
> access to the metadata path of the node being added or removed. This
> restricts the ability to use node A to manage node B, as node A may not
> have access to the metadata folder on node B.
> 2. Dependency on Node Configuration: The tool requires access to the
> configuration of the node being managed.
>
> > That said, I was thinking of an alternative:
> what if we change the logic so that *when both* --command-config
> *and* --controller-id are provided, the command config only applies
> to admin initialization? Users can add extra admin properties in
> --command-config.
>
> I'm good with it.
>
> >L04
> > As jose mentioned, the controller (observer) endpoints are stored in the
> ClusterImage
>
> Yes, I like this idea since we already have endpoint information stored in
> the active controller.
>
> Thank you.
> Luke
>
>
> On Fri, Aug 1, 2025 at 12:40 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
>
> > hi Kuan-Po
> >
> > Yeah, I agree — to make things easier, we could enhance the logic.
> > If the user provides --bootstrap-server, we can retrieve
> > controller.quorum.bootstrap.servers from the broker configs
> > using admin.describeConfigs, and use that to get the required
> > controller configs.
> >
> >
> > As jose mentioned, the controller (observer) endpoints are stored in the
> > ClusterImage.
> >
> > Another (more complicated) approach would be to make the endpoints in
> > AddRaftVoterRequest optional.
> >
> > Initially, I thought this KIP could address this issue in a simple way,
> > without requiring any changes to the RPC :)
> >
> > Best,
> > Chia-Ping
> >
> > Kuan-Po Tseng <br...@gmail.com> 於 2025年8月1日 週五 上午12:05寫道：
> >
> > > Hi Luke,
> > >
> > > Appreciate the feedback — glad to finally see some voices in this thread!
> > > :)
> > >
> > > *L01, L02:*
> > > Sure thing!
> > >
> > > *L03:*
> > > Thanks for the heads-up — I hadn’t thought of that before.
> > > In KAFKA-19563, you mentioned introducing a new
> > > --add-controller-command-config option to separate admin and
> > > controller configs. That sounds like a solid approach.
> > >
> > > That said, I was thinking of an alternative:
> > > what if we change the logic so that *when both* --command-config
> > > *and* --controller-id are provided, the command config only applies
> > > to admin initialization? Users can add extra admin properties in
> > > --command-config.
> > >
> > > What do you think?
> > >
> > > *L04:*
> > > Yeah, I agree — to make things easier, we could enhance the logic.
> > > If the user provides --bootstrap-server, we can retrieve
> > > controller.quorum.bootstrap.servers from the broker configs
> > > using admin.describeConfigs, and use that to get the required
> > > controller configs.
> > >
> > > Then we could allow users to run something like:
> > >
> > > bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 \
> > > add-controller --controller-id <id>
> > >
> > >
> > > Best,
> > > Kuan-Po
> > >
> > > On Thu, Jul 31, 2025 at 3:24 PM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > Hi Kuan-Po,
> > > >
> > > > Thanks for the KIP!
> > > > We also faced the similar issue recently.
> > > >
> > > > Some suggestions/questions:
> > > > 1. Could you also include the issue in KAFKA-19563
> > > > <https://issues.apache.org/jira/browse/KAFKA-19563> as the motivation
> > in
> > > > the KIP?
> > > > 2. Could you make it clear in the KIP that after the KIP, the
> > > > add-controller request from kafka-metadata-quorum.sh will take more
> > time
> > > > than before because of more API calls needed?
> > > > 3. "If —-command-config is provided, fallback to the existing behavior,
> > > and
> > > > —-controller-id option will be ignored."
> > > > On this, it looks like it still doesn't resolve the issue described in
> > > > KAFKA-19563 <https://issues.apache.org/jira/browse/KAFKA-19563>
> > because
> > > if
> > > > the client wants to pass the "client.id", and provide a properties
> > file
> > > in
> > > > --command-config, it 'll be treated as the controller.properties and
> > fail
> > > > the request, right? Ex:
> > > > `bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093
> > > > add-controller --controller-id <id> --command-config admin.properties`
> > > > 4. "This can only be done with bootstrap controller option since we
> > can‘t
> > > > use bootstrap.server in Admin#describeConfigs to get controller
> > configs."
> > > > I understand the reason but it makes the script more confusing in my
> > > > opinion.
> > > > I'm wondering why we need the controller endpoint when talking with the
> > > > broker with --bootstrap-server? The controller endpoint should already
> > > > exist in broker itself, right?
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Sat, Apr 19, 2025 at 11:32 PM Kuan Po Tseng <br...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi José,
> > > > >
> > > > > Apologies for the delayed response. Do you still have any questions
> > > > > regarding the describeConfig part? As Chia-Ping mentioned, we are
> > able
> > > to
> > > > > retrieve all broker configurations through that method. (
> > > > >
> > > >
> > >
> > https://github.com/apache/kafka/blob/da46cf6e79afbbed1da2bae831e0f70992e85f9b/core/src/main/scala/kafka/server/ConfigHelper.scala#L121-L123
> > > > > )
> > > > >
> > > > > Please feel free to reach out if you have any further questions or
> > need
> > > > > clarification. Thank you again for your valuable feedback!
> > > > >
> > > > > Best,
> > > > > Kuan-Po Tseng
> > > > >
> > > > > On 2025/04/03 16:04:36 José Armando García Sancio wrote:
> > > > > > Hi Chia,
> > > > > >
> > > > > > On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <
> > chia7712@apache.org>
> > > > > wrote:
> > > > > > > We propose to use `Admin#describeConfigs` to get the configs for
> > > > > specific controller if the bootstrap.controllers is configured. This
> > > > > approach is similar to what `MetadataQuorumCommand` does, and the
> > > > > difference is `MetadataQuorumCommand` read those configs from local
> > > file
> > > > > and this KIP gets those configs by `Admin#describeConfigs`
> > > > > >
> > > > > > I am not sure. I have to look at that code but doesn't
> > > > > > "Admin#describeConfigs" only return dynamic configuration for the
> > > > > > controller? Most users configure the controller using the server
> > > > > > properties file. My current understanding is that values coming
> > from
> > > > > > the properties file won't show up in Admin#describeConfigs.
> > > > > >
> > > > > > Thanks,
> > > > > > --
> > > > > > -José
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Luke Chen <sh...@gmail.com>.
Hi Kuan-Po,
> L03
> In KAFKA-19563, you mentioned introducing a new
--add-controller-command-config option to separate admin and
controller configs. That sounds like a solid approach.
Yes, that can resolve the issue in KAFKA-19563, but it conflicts with the
motivation in the KIP:
1. Limited Accessibility: The node executing the tool must have direct
access to the metadata path of the node being added or removed. This
restricts the ability to use node A to manage node B, as node A may not
have access to the metadata folder on node B.
2. Dependency on Node Configuration: The tool requires access to the
configuration of the node being managed.
> That said, I was thinking of an alternative:
what if we change the logic so that *when both* --command-config
*and* --controller-id are provided, the command config only applies
to admin initialization? Users can add extra admin properties in
--command-config.
I'm good with it.
>L04
> As jose mentioned, the controller (observer) endpoints are stored in the
ClusterImage
Yes, I like this idea since we already have endpoint information stored in
the active controller.
Thank you.
Luke
On Fri, Aug 1, 2025 at 12:40 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
> hi Kuan-Po
>
> Yeah, I agree — to make things easier, we could enhance the logic.
> If the user provides --bootstrap-server, we can retrieve
> controller.quorum.bootstrap.servers from the broker configs
> using admin.describeConfigs, and use that to get the required
> controller configs.
>
>
> As jose mentioned, the controller (observer) endpoints are stored in the
> ClusterImage.
>
> Another (more complicated) approach would be to make the endpoints in
> AddRaftVoterRequest optional.
>
> Initially, I thought this KIP could address this issue in a simple way,
> without requiring any changes to the RPC :)
>
> Best,
> Chia-Ping
>
> Kuan-Po Tseng <br...@gmail.com> 於 2025年8月1日 週五 上午12:05寫道：
>
> > Hi Luke,
> >
> > Appreciate the feedback — glad to finally see some voices in this thread!
> > :)
> >
> > *L01, L02:*
> > Sure thing!
> >
> > *L03:*
> > Thanks for the heads-up — I hadn’t thought of that before.
> > In KAFKA-19563, you mentioned introducing a new
> > --add-controller-command-config option to separate admin and
> > controller configs. That sounds like a solid approach.
> >
> > That said, I was thinking of an alternative:
> > what if we change the logic so that *when both* --command-config
> > *and* --controller-id are provided, the command config only applies
> > to admin initialization? Users can add extra admin properties in
> > --command-config.
> >
> > What do you think?
> >
> > *L04:*
> > Yeah, I agree — to make things easier, we could enhance the logic.
> > If the user provides --bootstrap-server, we can retrieve
> > controller.quorum.bootstrap.servers from the broker configs
> > using admin.describeConfigs, and use that to get the required
> > controller configs.
> >
> > Then we could allow users to run something like:
> >
> > bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 \
> > add-controller --controller-id <id>
> >
> >
> > Best,
> > Kuan-Po
> >
> > On Thu, Jul 31, 2025 at 3:24 PM Luke Chen <sh...@gmail.com> wrote:
> >
> > > Hi Kuan-Po,
> > >
> > > Thanks for the KIP!
> > > We also faced the similar issue recently.
> > >
> > > Some suggestions/questions:
> > > 1. Could you also include the issue in KAFKA-19563
> > > <https://issues.apache.org/jira/browse/KAFKA-19563> as the motivation
> in
> > > the KIP?
> > > 2. Could you make it clear in the KIP that after the KIP, the
> > > add-controller request from kafka-metadata-quorum.sh will take more
> time
> > > than before because of more API calls needed?
> > > 3. "If —-command-config is provided, fallback to the existing behavior,
> > and
> > > —-controller-id option will be ignored."
> > > On this, it looks like it still doesn't resolve the issue described in
> > > KAFKA-19563 <https://issues.apache.org/jira/browse/KAFKA-19563>
> because
> > if
> > > the client wants to pass the "client.id", and provide a properties
> file
> > in
> > > --command-config, it 'll be treated as the controller.properties and
> fail
> > > the request, right? Ex:
> > > `bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093
> > > add-controller --controller-id <id> --command-config admin.properties`
> > > 4. "This can only be done with bootstrap controller option since we
> can‘t
> > > use bootstrap.server in Admin#describeConfigs to get controller
> configs."
> > > I understand the reason but it makes the script more confusing in my
> > > opinion.
> > > I'm wondering why we need the controller endpoint when talking with the
> > > broker with --bootstrap-server? The controller endpoint should already
> > > exist in broker itself, right?
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Sat, Apr 19, 2025 at 11:32 PM Kuan Po Tseng <br...@apache.org>
> > > wrote:
> > >
> > > > Hi José,
> > > >
> > > > Apologies for the delayed response. Do you still have any questions
> > > > regarding the describeConfig part? As Chia-Ping mentioned, we are
> able
> > to
> > > > retrieve all broker configurations through that method. (
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/da46cf6e79afbbed1da2bae831e0f70992e85f9b/core/src/main/scala/kafka/server/ConfigHelper.scala#L121-L123
> > > > )
> > > >
> > > > Please feel free to reach out if you have any further questions or
> need
> > > > clarification. Thank you again for your valuable feedback!
> > > >
> > > > Best,
> > > > Kuan-Po Tseng
> > > >
> > > > On 2025/04/03 16:04:36 José Armando García Sancio wrote:
> > > > > Hi Chia,
> > > > >
> > > > > On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <
> chia7712@apache.org>
> > > > wrote:
> > > > > > We propose to use `Admin#describeConfigs` to get the configs for
> > > > specific controller if the bootstrap.controllers is configured. This
> > > > approach is similar to what `MetadataQuorumCommand` does, and the
> > > > difference is `MetadataQuorumCommand` read those configs from local
> > file
> > > > and this KIP gets those configs by `Admin#describeConfigs`
> > > > >
> > > > > I am not sure. I have to look at that code but doesn't
> > > > > "Admin#describeConfigs" only return dynamic configuration for the
> > > > > controller? Most users configure the controller using the server
> > > > > properties file. My current understanding is that values coming
> from
> > > > > the properties file won't show up in Admin#describeConfigs.
> > > > >
> > > > > Thanks,
> > > > > --
> > > > > -José
> > > > >
> > > >
> > >
> >
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi Kuan-Po
Yeah, I agree — to make things easier, we could enhance the logic.
If the user provides --bootstrap-server, we can retrieve
controller.quorum.bootstrap.servers from the broker configs
using admin.describeConfigs, and use that to get the required
controller configs.
As jose mentioned, the controller (observer) endpoints are stored in the
ClusterImage.
Another (more complicated) approach would be to make the endpoints in
AddRaftVoterRequest optional.
Initially, I thought this KIP could address this issue in a simple way,
without requiring any changes to the RPC :)
Best,
Chia-Ping
Kuan-Po Tseng <br...@gmail.com> 於 2025年8月1日 週五 上午12:05寫道：
> Hi Luke,
>
> Appreciate the feedback — glad to finally see some voices in this thread!
> :)
>
> *L01, L02:*
> Sure thing!
>
> *L03:*
> Thanks for the heads-up — I hadn’t thought of that before.
> In KAFKA-19563, you mentioned introducing a new
> --add-controller-command-config option to separate admin and
> controller configs. That sounds like a solid approach.
>
> That said, I was thinking of an alternative:
> what if we change the logic so that *when both* --command-config
> *and* --controller-id are provided, the command config only applies
> to admin initialization? Users can add extra admin properties in
> --command-config.
>
> What do you think?
>
> *L04:*
> Yeah, I agree — to make things easier, we could enhance the logic.
> If the user provides --bootstrap-server, we can retrieve
> controller.quorum.bootstrap.servers from the broker configs
> using admin.describeConfigs, and use that to get the required
> controller configs.
>
> Then we could allow users to run something like:
>
> bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 \
> add-controller --controller-id <id>
>
>
> Best,
> Kuan-Po
>
> On Thu, Jul 31, 2025 at 3:24 PM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Kuan-Po,
> >
> > Thanks for the KIP!
> > We also faced the similar issue recently.
> >
> > Some suggestions/questions:
> > 1. Could you also include the issue in KAFKA-19563
> > <https://issues.apache.org/jira/browse/KAFKA-19563> as the motivation in
> > the KIP?
> > 2. Could you make it clear in the KIP that after the KIP, the
> > add-controller request from kafka-metadata-quorum.sh will take more time
> > than before because of more API calls needed?
> > 3. "If —-command-config is provided, fallback to the existing behavior,
> and
> > —-controller-id option will be ignored."
> > On this, it looks like it still doesn't resolve the issue described in
> > KAFKA-19563 <https://issues.apache.org/jira/browse/KAFKA-19563> because
> if
> > the client wants to pass the "client.id", and provide a properties file
> in
> > --command-config, it 'll be treated as the controller.properties and fail
> > the request, right? Ex:
> > `bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093
> > add-controller --controller-id <id> --command-config admin.properties`
> > 4. "This can only be done with bootstrap controller option since we can‘t
> > use bootstrap.server in Admin#describeConfigs to get controller configs."
> > I understand the reason but it makes the script more confusing in my
> > opinion.
> > I'm wondering why we need the controller endpoint when talking with the
> > broker with --bootstrap-server? The controller endpoint should already
> > exist in broker itself, right?
> >
> > Thank you.
> > Luke
> >
> > On Sat, Apr 19, 2025 at 11:32 PM Kuan Po Tseng <br...@apache.org>
> > wrote:
> >
> > > Hi José,
> > >
> > > Apologies for the delayed response. Do you still have any questions
> > > regarding the describeConfig part? As Chia-Ping mentioned, we are able
> to
> > > retrieve all broker configurations through that method. (
> > >
> >
> https://github.com/apache/kafka/blob/da46cf6e79afbbed1da2bae831e0f70992e85f9b/core/src/main/scala/kafka/server/ConfigHelper.scala#L121-L123
> > > )
> > >
> > > Please feel free to reach out if you have any further questions or need
> > > clarification. Thank you again for your valuable feedback!
> > >
> > > Best,
> > > Kuan-Po Tseng
> > >
> > > On 2025/04/03 16:04:36 José Armando García Sancio wrote:
> > > > Hi Chia,
> > > >
> > > > On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <ch...@apache.org>
> > > wrote:
> > > > > We propose to use `Admin#describeConfigs` to get the configs for
> > > specific controller if the bootstrap.controllers is configured. This
> > > approach is similar to what `MetadataQuorumCommand` does, and the
> > > difference is `MetadataQuorumCommand` read those configs from local
> file
> > > and this KIP gets those configs by `Admin#describeConfigs`
> > > >
> > > > I am not sure. I have to look at that code but doesn't
> > > > "Admin#describeConfigs" only return dynamic configuration for the
> > > > controller? Most users configure the controller using the server
> > > > properties file. My current understanding is that values coming from
> > > > the properties file won't show up in Admin#describeConfigs.
> > > >
> > > > Thanks,
> > > > --
> > > > -José
> > > >
> > >
> >
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Kuan-Po Tseng <br...@gmail.com>.
Hi Luke,
Appreciate the feedback — glad to finally see some voices in this thread! :)
*L01, L02:*
Sure thing!
*L03:*
Thanks for the heads-up — I hadn’t thought of that before.
In KAFKA-19563, you mentioned introducing a new
--add-controller-command-config option to separate admin and
controller configs. That sounds like a solid approach.
That said, I was thinking of an alternative:
what if we change the logic so that *when both* --command-config
*and* --controller-id are provided, the command config only applies
to admin initialization? Users can add extra admin properties in
--command-config.
What do you think?
*L04:*
Yeah, I agree — to make things easier, we could enhance the logic.
If the user provides --bootstrap-server, we can retrieve
controller.quorum.bootstrap.servers from the broker configs
using admin.describeConfigs, and use that to get the required
controller configs.
Then we could allow users to run something like:
bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 \
add-controller --controller-id <id>
Best,
Kuan-Po
On Thu, Jul 31, 2025 at 3:24 PM Luke Chen <sh...@gmail.com> wrote:
> Hi Kuan-Po,
>
> Thanks for the KIP!
> We also faced the similar issue recently.
>
> Some suggestions/questions:
> 1. Could you also include the issue in KAFKA-19563
> <https://issues.apache.org/jira/browse/KAFKA-19563> as the motivation in
> the KIP?
> 2. Could you make it clear in the KIP that after the KIP, the
> add-controller request from kafka-metadata-quorum.sh will take more time
> than before because of more API calls needed?
> 3. "If —-command-config is provided, fallback to the existing behavior, and
> —-controller-id option will be ignored."
> On this, it looks like it still doesn't resolve the issue described in
> KAFKA-19563 <https://issues.apache.org/jira/browse/KAFKA-19563> because if
> the client wants to pass the "client.id", and provide a properties file in
> --command-config, it 'll be treated as the controller.properties and fail
> the request, right? Ex:
> `bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093
> add-controller --controller-id <id> --command-config admin.properties`
> 4. "This can only be done with bootstrap controller option since we can‘t
> use bootstrap.server in Admin#describeConfigs to get controller configs."
> I understand the reason but it makes the script more confusing in my
> opinion.
> I'm wondering why we need the controller endpoint when talking with the
> broker with --bootstrap-server? The controller endpoint should already
> exist in broker itself, right?
>
> Thank you.
> Luke
>
> On Sat, Apr 19, 2025 at 11:32 PM Kuan Po Tseng <br...@apache.org>
> wrote:
>
> > Hi José,
> >
> > Apologies for the delayed response. Do you still have any questions
> > regarding the describeConfig part? As Chia-Ping mentioned, we are able to
> > retrieve all broker configurations through that method. (
> >
> https://github.com/apache/kafka/blob/da46cf6e79afbbed1da2bae831e0f70992e85f9b/core/src/main/scala/kafka/server/ConfigHelper.scala#L121-L123
> > )
> >
> > Please feel free to reach out if you have any further questions or need
> > clarification. Thank you again for your valuable feedback!
> >
> > Best,
> > Kuan-Po Tseng
> >
> > On 2025/04/03 16:04:36 José Armando García Sancio wrote:
> > > Hi Chia,
> > >
> > > On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> > > > We propose to use `Admin#describeConfigs` to get the configs for
> > specific controller if the bootstrap.controllers is configured. This
> > approach is similar to what `MetadataQuorumCommand` does, and the
> > difference is `MetadataQuorumCommand` read those configs from local file
> > and this KIP gets those configs by `Admin#describeConfigs`
> > >
> > > I am not sure. I have to look at that code but doesn't
> > > "Admin#describeConfigs" only return dynamic configuration for the
> > > controller? Most users configure the controller using the server
> > > properties file. My current understanding is that values coming from
> > > the properties file won't show up in Admin#describeConfigs.
> > >
> > > Thanks,
> > > --
> > > -José
> > >
> >
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Luke Chen <sh...@gmail.com>.
Hi Kuan-Po,
Thanks for the KIP!
We also faced the similar issue recently.
Some suggestions/questions:
1. Could you also include the issue in KAFKA-19563
<https://issues.apache.org/jira/browse/KAFKA-19563> as the motivation in
the KIP?
2. Could you make it clear in the KIP that after the KIP, the
add-controller request from kafka-metadata-quorum.sh will take more time
than before because of more API calls needed?
3. "If —-command-config is provided, fallback to the existing behavior, and
—-controller-id option will be ignored."
On this, it looks like it still doesn't resolve the issue described in
KAFKA-19563 <https://issues.apache.org/jira/browse/KAFKA-19563> because if
the client wants to pass the "client.id", and provide a properties file in
--command-config, it 'll be treated as the controller.properties and fail
the request, right? Ex:
`bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093
add-controller --controller-id <id> --command-config admin.properties`
4. "This can only be done with bootstrap controller option since we can‘t
use bootstrap.server in Admin#describeConfigs to get controller configs."
I understand the reason but it makes the script more confusing in my
opinion.
I'm wondering why we need the controller endpoint when talking with the
broker with --bootstrap-server? The controller endpoint should already
exist in broker itself, right?
Thank you.
Luke
On Sat, Apr 19, 2025 at 11:32 PM Kuan Po Tseng <br...@apache.org> wrote:
> Hi José,
>
> Apologies for the delayed response. Do you still have any questions
> regarding the describeConfig part? As Chia-Ping mentioned, we are able to
> retrieve all broker configurations through that method. (
> https://github.com/apache/kafka/blob/da46cf6e79afbbed1da2bae831e0f70992e85f9b/core/src/main/scala/kafka/server/ConfigHelper.scala#L121-L123
> )
>
> Please feel free to reach out if you have any further questions or need
> clarification. Thank you again for your valuable feedback!
>
> Best,
> Kuan-Po Tseng
>
> On 2025/04/03 16:04:36 José Armando García Sancio wrote:
> > Hi Chia,
> >
> > On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <ch...@apache.org>
> wrote:
> > > We propose to use `Admin#describeConfigs` to get the configs for
> specific controller if the bootstrap.controllers is configured. This
> approach is similar to what `MetadataQuorumCommand` does, and the
> difference is `MetadataQuorumCommand` read those configs from local file
> and this KIP gets those configs by `Admin#describeConfigs`
> >
> > I am not sure. I have to look at that code but doesn't
> > "Admin#describeConfigs" only return dynamic configuration for the
> > controller? Most users configure the controller using the server
> > properties file. My current understanding is that values coming from
> > the properties file won't show up in Admin#describeConfigs.
> >
> > Thanks,
> > --
> > -José
> >
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Kuan Po Tseng <br...@apache.org>.
Hi José,
Apologies for the delayed response. Do you still have any questions regarding the describeConfig part? As Chia-Ping mentioned, we are able to retrieve all broker configurations through that method. (https://github.com/apache/kafka/blob/da46cf6e79afbbed1da2bae831e0f70992e85f9b/core/src/main/scala/kafka/server/ConfigHelper.scala#L121-L123)
Please feel free to reach out if you have any further questions or need clarification. Thank you again for your valuable feedback!
Best,
Kuan-Po Tseng
On 2025/04/03 16:04:36 José Armando García Sancio wrote:
> Hi Chia,
>
> On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> > We propose to use `Admin#describeConfigs` to get the configs for specific controller if the bootstrap.controllers is configured. This approach is similar to what `MetadataQuorumCommand` does, and the difference is `MetadataQuorumCommand` read those configs from local file and this KIP gets those configs by `Admin#describeConfigs`
>
> I am not sure. I have to look at that code but doesn't
> "Admin#describeConfigs" only return dynamic configuration for the
> controller? Most users configure the controller using the server
> properties file. My current understanding is that values coming from
> the properties file won't show up in Admin#describeConfigs.
>
> Thanks,
> --
> -José
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi Jose
> I am not sure. I have to look at that code but doesn't
"Admin#describeConfigs" only return dynamic configuration for the
controller? Most users configure the controller using the server
properties file. My current understanding is that values coming from
the properties file won't show up in Admin#describeConfigs.
The returned DescribeConfigsResult include both dynamic and static (from
properties) configs, so we can get both listeners and advertised.listeners
from `Admin#describeConfigs`.
see
https://github.com/apache/kafka/blob/trunk/clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/StaticBrokerConfigTest.java#L57
for example
Best,
Chia-Ping
José Armando García Sancio <js...@confluent.io.invalid> 於 2025年4月4日 週五
上午12:05寫道：
> Hi Chia,
>
> On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <ch...@apache.org>
> wrote:
> > We propose to use `Admin#describeConfigs` to get the configs for
> specific controller if the bootstrap.controllers is configured. This
> approach is similar to what `MetadataQuorumCommand` does, and the
> difference is `MetadataQuorumCommand` read those configs from local file
> and this KIP gets those configs by `Admin#describeConfigs`
>
> I am not sure. I have to look at that code but doesn't
> "Admin#describeConfigs" only return dynamic configuration for the
> controller? Most users configure the controller using the server
> properties file. My current understanding is that values coming from
> the properties file won't show up in Admin#describeConfigs.
>
> Thanks,
> --
> -José
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by José Armando García Sancio <js...@confluent.io.INVALID>.
Hi Chia,
On Thu, Apr 3, 2025 at 10:24 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> We propose to use `Admin#describeConfigs` to get the configs for specific controller if the bootstrap.controllers is configured. This approach is similar to what `MetadataQuorumCommand` does, and the difference is `MetadataQuorumCommand` read those configs from local file and this KIP gets those configs by `Admin#describeConfigs`
I am not sure. I have to look at that code but doesn't
"Admin#describeConfigs" only return dynamic configuration for the
controller? Most users configure the controller using the server
properties file. My current understanding is that values coming from
the properties file won't show up in Admin#describeConfigs.
Thanks,
--
-José
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Jose
Thanks for sharing. I have a question.
> To my knowledge this information is not available to the Admin client.
We propose to use `Admin#describeConfigs` to get the configs for specific controller if the bootstrap.controllers is configured. This approach is similar to what `MetadataQuorumCommand` does, and the difference is `MetadataQuorumCommand` read those configs from local file and this KIP gets those configs by `Admin#describeConfigs`
Please correct me if I misunderstand your point.
Best,
Chia-Ping
On 2025/04/02 21:04:27 José Armando García Sancio wrote:
> Hi Kuan,
>
> I took a look at your KIP and thought about the problem and possible solutions.
>
> For the add-controller command, the active controller needs to know
> the new controller's id, directory id and endpoints for the default
> listener. As you mentioned in your KIP the user can provide the id
> using the --controller-id flag. This seems reasonable since this is a
> value that they generate and understand. For directory id, I think we
> can make it optional in the RPC but the kraft leader can look it up in
> the in-memory state stored for all replicas (observers). For the
> endpoint, things get tricky. This needs to be an endpoint that the
> network client used by KRaft knows how to connect and authenticate
> against. The network client used by kraft is configured to use the
> protocol described for the first listener in
> controller.listener.names.
>
> To my knowledge this information is not available to the Admin client.
> If we want to make this information available to the Admin client, I
> think that the best RPC to do that is the DescribeCluster RPC.
> Unfortunately, that RPC only returns the endpoints that are associated
> with the port (listener) that was used to contact the bootstrap
> controller (--bootstrap-controller) which may not be the endpoint
> configured for the first listener in controller.listener.names. The
> other issue is that the DescribeCluster RPC doesn't return controller
> endpoints if the bootstrap server (--bootstrap-server) is used. The
> DescribeCluster RPC can be extended so that it returns all endpoints
> for all nodes (controller and brokers).
>
> Another way to solve this, is to make the endpoints optional. In the
> current implementation all controllers send a controller registration
> request to the active controller even if they are not voters (cannot
> become active controller). This information is stored in the
> ClusterImage. The endpoints stored in ClusterImage can be used to
> populate the request sent to KafkaRaftClient.
>
> For the remove-controller command, the active controller needs to know
> the old controller's id and directory id. We can make the directory id
> optional. I think this is okay since kraft guarantees that id is
> unique in the voter set. Not specified the directory id is not as safe
> and idempotent but it is up to the user to opt in. We can make that
> clear by the description for remove-controller and --controller-uuid.
>
> What do you think?
> --
> -José
>
Re: KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management
Posted by José Armando García Sancio <js...@confluent.io.INVALID>.
Hi Kuan,
I took a look at your KIP and thought about the problem and possible solutions.
For the add-controller command, the active controller needs to know
the new controller's id, directory id and endpoints for the default
listener. As you mentioned in your KIP the user can provide the id
using the --controller-id flag. This seems reasonable since this is a
value that they generate and understand. For directory id, I think we
can make it optional in the RPC but the kraft leader can look it up in
the in-memory state stored for all replicas (observers). For the
endpoint, things get tricky. This needs to be an endpoint that the
network client used by KRaft knows how to connect and authenticate
against. The network client used by kraft is configured to use the
protocol described for the first listener in
controller.listener.names.
To my knowledge this information is not available to the Admin client.
If we want to make this information available to the Admin client, I
think that the best RPC to do that is the DescribeCluster RPC.
Unfortunately, that RPC only returns the endpoints that are associated
with the port (listener) that was used to contact the bootstrap
controller (--bootstrap-controller) which may not be the endpoint
configured for the first listener in controller.listener.names. The
other issue is that the DescribeCluster RPC doesn't return controller
endpoints if the bootstrap server (--bootstrap-server) is used. The
DescribeCluster RPC can be extended so that it returns all endpoints
for all nodes (controller and brokers).
Another way to solve this, is to make the endpoints optional. In the
current implementation all controllers send a controller registration
request to the active controller even if they are not voters (cannot
become active controller). This information is stored in the
ClusterImage. The endpoints stored in ClusterImage can be used to
populate the request sent to KafkaRaftClient.
For the remove-controller command, the active controller needs to know
the old controller's id and directory id. We can make the directory id
optional. I think this is okay since kraft guarantees that id is
unique in the voter set. Not specified the directory id is not as safe
and idempotent but it is up to the user to opt in. We can make that
clear by the description for remove-controller and --controller-uuid.
What do you think?
--
-José