You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Kuan Po Tseng <br...@gmail.com> on 2025/04/22 15:56:53 UTC

[VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Hello everyone,

I would like to start a vote on KIP-1141, thank you!

KIP: https://cwiki.apache.org/confluence/x/PZFEF
Discussion thread:
https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4

Best,
Kuan-Po Tseng

Re: [VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Posted by PoAn Yang <ya...@gmail.com>.

Thanks for the KIP.

+1 (binding)

> On Mar 12, 2026, at 6:09 PM, Chia-Ping Tsai <ch...@gmail.com> wrote:
> 
> +1 (binding)
>> Luke Chen <sh...@gmail.com> 於 2026年3月12日 下午3:08 寫道：
>> 
>> I had another look. LGTM!
>> 
>> Thanks for the KIP.
>> +1 (binding) from me.
>> 
>> Thank you,
>> Luke
>> 
>>> On Tue, Apr 29, 2025 at 12:59 PM Kuan Po Tseng <br...@gmail.com> wrote:
>>> 
>>> Hi Colin,
>>> 
>>> I just wanted to make sure I’m on the right track, so I’m sharing my
>>> understanding of
>>> the current process for adding a controller to the voter set.
>>> 
>>> Here's what I think it looks like:
>>> Step 1: Use kafka-storage.sh format to format the controller.
>>> Step 2: Start up the controller and ensure it starts up successfully.
>>> Step 3: Provide the controller config and use the kafka-metadata-quorum.sh
>>> add-controller command to add the controller to the voter set.
>>> 
>>> The KIP aims to simplify "Step 3". It removes the need to provide the
>>> controller config,
>>> as all the information that add-controller requires—like the directory UUID
>>> and controller endpoint—
>>> can be gathered through Admin#describeMetadataQuorum and
>>> Admin#describeConfigs.
>>> Since the controller is already in the cluster by Step 2, this information
>>> should be available.
>>> 
>>> So, users would still need to complete "Step 1" and "Step 2".
>>> Otherwise, as you mentioned, we won’t be able to retrieve the controller
>>> UUID and its corresponding endpoints for Step 3.
>>> 
>>> I’m not sure if I’ve misunderstood anything, so please let me know if
>>> there’s anything I’ve missed.
>>> 
>>> Thanks so much for reviewing my KIP!
>>> 
>>> Best regards,
>>> Kuan-Po Tseng
>>> 
>>>> On Tue, Apr 29, 2025 at 8:50 AM Colin McCabe <cm...@apache.org> wrote:
>>>> 
>>>> So to expand a bit on what I wrote below:
>>>> 
>>>> - If the goal is to make it possible to have adding the controller node
>>>> happen in one place, and formatting the controller node happen in
>>> another,
>>>> those two places still need to communicate. What they need to communicate
>>>> is the new directory ID. This is basically a random UUID that must never
>>> be
>>>> reused.
>>>> 
>>>> - The most obvious way to do that is to format the new controller node
>>>> first, and then grab that ID. This is what the current system does, by
>>>> reading that local file.
>>>> 
>>>> - Another way to do it would be to generate the UUID somewhere else, and
>>>> then format using --initial-controllers. This currently requires spelling
>>>> out the full set of controllers and matching directory ids in the
>>> cluster.
>>>> One thing that is a bit subtle is that the tool will actually look at
>>> what
>>>> directory ID you gave, and use that directory ID when formatting your
>>>> directory. This should be documented a bit more prominently if we want
>>>> people to use this mechanism.
>>>> 
>>>> best,
>>>> Colin
>>>> 
>>>> 
>>>> On Mon, Apr 28, 2025, at 13:21, Colin McCabe wrote:
>>>>> Hi Kuan Po Tseng,
>>>>> 
>>>>> It is not possible to add a new controller purely by ID. You need
>>>>> controller ID, directory ID, and controller endpoints.
>>>>> 
>>>>> These cannot be obtained through describeMetadataQuorum since that only
>>>>> describes the existing quorum, not what you would like the new one to
>>>>> be. Also, obviously, we never want to reuse an existing directory ID.
>>>>> 
>>>>> Therefore I will vote -1 on this pending further discussion.
>>>>> 
>>>>> best,
>>>>> Colin
>>>>> 
>>>>> On Tue, Apr 22, 2025, at 08:56, Kuan Po Tseng wrote:
>>>>>> Hello everyone,
>>>>>> 
>>>>>> I would like to start a vote on KIP-1141, thank you!
>>>>>> 
>>>>>> KIP: https://cwiki.apache.org/confluence/x/PZFEF
>>>>>> Discussion thread:
>>>>>> https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4
>>>>>> 
>>>>>> Best,
>>>>>> Kuan-Po Tseng
>>>> 
>>>

Re: [VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Posted by Chia-Ping Tsai <ch...@gmail.com>.

+1 (binding)
> Luke Chen <sh...@gmail.com> 於 2026年3月12日 下午3:08 寫道：
> 
> I had another look. LGTM!
> 
> Thanks for the KIP.
> +1 (binding) from me.
> 
> Thank you,
> Luke
> 
>> On Tue, Apr 29, 2025 at 12:59 PM Kuan Po Tseng <br...@gmail.com> wrote:
>> 
>> Hi Colin,
>> 
>> I just wanted to make sure I’m on the right track, so I’m sharing my
>> understanding of
>> the current process for adding a controller to the voter set.
>> 
>> Here's what I think it looks like:
>> Step 1: Use kafka-storage.sh format to format the controller.
>> Step 2: Start up the controller and ensure it starts up successfully.
>> Step 3: Provide the controller config and use the kafka-metadata-quorum.sh
>> add-controller command to add the controller to the voter set.
>> 
>> The KIP aims to simplify "Step 3". It removes the need to provide the
>> controller config,
>> as all the information that add-controller requires—like the directory UUID
>> and controller endpoint—
>> can be gathered through Admin#describeMetadataQuorum and
>> Admin#describeConfigs.
>> Since the controller is already in the cluster by Step 2, this information
>> should be available.
>> 
>> So, users would still need to complete "Step 1" and "Step 2".
>> Otherwise, as you mentioned, we won’t be able to retrieve the controller
>> UUID and its corresponding endpoints for Step 3.
>> 
>> I’m not sure if I’ve misunderstood anything, so please let me know if
>> there’s anything I’ve missed.
>> 
>> Thanks so much for reviewing my KIP!
>> 
>> Best regards,
>> Kuan-Po Tseng
>> 
>>> On Tue, Apr 29, 2025 at 8:50 AM Colin McCabe <cm...@apache.org> wrote:
>>> 
>>> So to expand a bit on what I wrote below:
>>> 
>>> - If the goal is to make it possible to have adding the controller node
>>> happen in one place, and formatting the controller node happen in
>> another,
>>> those two places still need to communicate. What they need to communicate
>>> is the new directory ID. This is basically a random UUID that must never
>> be
>>> reused.
>>> 
>>> - The most obvious way to do that is to format the new controller node
>>> first, and then grab that ID. This is what the current system does, by
>>> reading that local file.
>>> 
>>> - Another way to do it would be to generate the UUID somewhere else, and
>>> then format using --initial-controllers. This currently requires spelling
>>> out the full set of controllers and matching directory ids in the
>> cluster.
>>> One thing that is a bit subtle is that the tool will actually look at
>> what
>>> directory ID you gave, and use that directory ID when formatting your
>>> directory. This should be documented a bit more prominently if we want
>>> people to use this mechanism.
>>> 
>>> best,
>>> Colin
>>> 
>>> 
>>> On Mon, Apr 28, 2025, at 13:21, Colin McCabe wrote:
>>>> Hi Kuan Po Tseng,
>>>> 
>>>> It is not possible to add a new controller purely by ID. You need
>>>> controller ID, directory ID, and controller endpoints.
>>>> 
>>>> These cannot be obtained through describeMetadataQuorum since that only
>>>> describes the existing quorum, not what you would like the new one to
>>>> be. Also, obviously, we never want to reuse an existing directory ID.
>>>> 
>>>> Therefore I will vote -1 on this pending further discussion.
>>>> 
>>>> best,
>>>> Colin
>>>> 
>>>> On Tue, Apr 22, 2025, at 08:56, Kuan Po Tseng wrote:
>>>>> Hello everyone,
>>>>> 
>>>>> I would like to start a vote on KIP-1141, thank you!
>>>>> 
>>>>> KIP: https://cwiki.apache.org/confluence/x/PZFEF
>>>>> Discussion thread:
>>>>> https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4
>>>>> 
>>>>> Best,
>>>>> Kuan-Po Tseng
>>> 
>>

Re: [VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Posted by Luke Chen <sh...@gmail.com>.

I had another look. LGTM!

Thanks for the KIP.
+1 (binding) from me.

Thank you,
Luke

On Tue, Apr 29, 2025 at 12:59 PM Kuan Po Tseng <br...@gmail.com> wrote:

> Hi Colin,
>
> I just wanted to make sure I’m on the right track, so I’m sharing my
> understanding of
> the current process for adding a controller to the voter set.
>
> Here's what I think it looks like:
> Step 1: Use kafka-storage.sh format to format the controller.
> Step 2: Start up the controller and ensure it starts up successfully.
> Step 3: Provide the controller config and use the kafka-metadata-quorum.sh
> add-controller command to add the controller to the voter set.
>
> The KIP aims to simplify "Step 3". It removes the need to provide the
> controller config,
> as all the information that add-controller requires—like the directory UUID
> and controller endpoint—
> can be gathered through Admin#describeMetadataQuorum and
> Admin#describeConfigs.
> Since the controller is already in the cluster by Step 2, this information
> should be available.
>
> So, users would still need to complete "Step 1" and "Step 2".
> Otherwise, as you mentioned, we won’t be able to retrieve the controller
> UUID and its corresponding endpoints for Step 3.
>
> I’m not sure if I’ve misunderstood anything, so please let me know if
> there’s anything I’ve missed.
>
> Thanks so much for reviewing my KIP!
>
> Best regards,
> Kuan-Po Tseng
>
> On Tue, Apr 29, 2025 at 8:50 AM Colin McCabe <cm...@apache.org> wrote:
>
> > So to expand a bit on what I wrote below:
> >
> > - If the goal is to make it possible to have adding the controller node
> > happen in one place, and formatting the controller node happen in
> another,
> > those two places still need to communicate. What they need to communicate
> > is the new directory ID. This is basically a random UUID that must never
> be
> > reused.
> >
> > - The most obvious way to do that is to format the new controller node
> > first, and then grab that ID. This is what the current system does, by
> > reading that local file.
> >
> > - Another way to do it would be to generate the UUID somewhere else, and
> > then format using --initial-controllers. This currently requires spelling
> > out the full set of controllers and matching directory ids in the
> cluster.
> > One thing that is a bit subtle is that the tool will actually look at
> what
> > directory ID you gave, and use that directory ID when formatting your
> > directory. This should be documented a bit more prominently if we want
> > people to use this mechanism.
> >
> > best,
> > Colin
> >
> >
> > On Mon, Apr 28, 2025, at 13:21, Colin McCabe wrote:
> > > Hi Kuan Po Tseng,
> > >
> > > It is not possible to add a new controller purely by ID. You need
> > > controller ID, directory ID, and controller endpoints.
> > >
> > > These cannot be obtained through describeMetadataQuorum since that only
> > > describes the existing quorum, not what you would like the new one to
> > > be. Also, obviously, we never want to reuse an existing directory ID.
> > >
> > > Therefore I will vote -1 on this pending further discussion.
> > >
> > > best,
> > > Colin
> > >
> > > On Tue, Apr 22, 2025, at 08:56, Kuan Po Tseng wrote:
> > >> Hello everyone,
> > >>
> > >> I would like to start a vote on KIP-1141, thank you!
> > >>
> > >> KIP: https://cwiki.apache.org/confluence/x/PZFEF
> > >> Discussion thread:
> > >> https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4
> > >>
> > >> Best,
> > >> Kuan-Po Tseng
> >
>

Re: [VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Posted by Kuan Po Tseng <br...@gmail.com>.

Hi Colin,

I just wanted to make sure I’m on the right track, so I’m sharing my
understanding of
the current process for adding a controller to the voter set.

Here's what I think it looks like:
Step 1: Use kafka-storage.sh format to format the controller.
Step 2: Start up the controller and ensure it starts up successfully.
Step 3: Provide the controller config and use the kafka-metadata-quorum.sh
add-controller command to add the controller to the voter set.

The KIP aims to simplify "Step 3". It removes the need to provide the
controller config,
as all the information that add-controller requires—like the directory UUID
and controller endpoint—
can be gathered through Admin#describeMetadataQuorum and
Admin#describeConfigs.
Since the controller is already in the cluster by Step 2, this information
should be available.

So, users would still need to complete "Step 1" and "Step 2".
Otherwise, as you mentioned, we won’t be able to retrieve the controller
UUID and its corresponding endpoints for Step 3.

I’m not sure if I’ve misunderstood anything, so please let me know if
there’s anything I’ve missed.

Thanks so much for reviewing my KIP!

Best regards,
Kuan-Po Tseng

On Tue, Apr 29, 2025 at 8:50 AM Colin McCabe <cm...@apache.org> wrote:

> So to expand a bit on what I wrote below:
>
> - If the goal is to make it possible to have adding the controller node
> happen in one place, and formatting the controller node happen in another,
> those two places still need to communicate. What they need to communicate
> is the new directory ID. This is basically a random UUID that must never be
> reused.
>
> - The most obvious way to do that is to format the new controller node
> first, and then grab that ID. This is what the current system does, by
> reading that local file.
>
> - Another way to do it would be to generate the UUID somewhere else, and
> then format using --initial-controllers. This currently requires spelling
> out the full set of controllers and matching directory ids in the cluster.
> One thing that is a bit subtle is that the tool will actually look at what
> directory ID you gave, and use that directory ID when formatting your
> directory. This should be documented a bit more prominently if we want
> people to use this mechanism.
>
> best,
> Colin
>
>
> On Mon, Apr 28, 2025, at 13:21, Colin McCabe wrote:
> > Hi Kuan Po Tseng,
> >
> > It is not possible to add a new controller purely by ID. You need
> > controller ID, directory ID, and controller endpoints.
> >
> > These cannot be obtained through describeMetadataQuorum since that only
> > describes the existing quorum, not what you would like the new one to
> > be. Also, obviously, we never want to reuse an existing directory ID.
> >
> > Therefore I will vote -1 on this pending further discussion.
> >
> > best,
> > Colin
> >
> > On Tue, Apr 22, 2025, at 08:56, Kuan Po Tseng wrote:
> >> Hello everyone,
> >>
> >> I would like to start a vote on KIP-1141, thank you!
> >>
> >> KIP: https://cwiki.apache.org/confluence/x/PZFEF
> >> Discussion thread:
> >> https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4
> >>
> >> Best,
> >> Kuan-Po Tseng
>

Re: [VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Posted by Colin McCabe <cm...@apache.org>.

So to expand a bit on what I wrote below:

- If the goal is to make it possible to have adding the controller node happen in one place, and formatting the controller node happen in another, those two places still need to communicate. What they need to communicate is the new directory ID. This is basically a random UUID that must never be reused.

- The most obvious way to do that is to format the new controller node first, and then grab that ID. This is what the current system does, by reading that local file.

- Another way to do it would be to generate the UUID somewhere else, and then format using --initial-controllers. This currently requires spelling out the full set of controllers and matching directory ids in the cluster. One thing that is a bit subtle is that the tool will actually look at what directory ID you gave, and use that directory ID when formatting your directory. This should be documented a bit more prominently if we want people to use this mechanism.

best,
Colin


On Mon, Apr 28, 2025, at 13:21, Colin McCabe wrote:
> Hi Kuan Po Tseng,
>
> It is not possible to add a new controller purely by ID. You need 
> controller ID, directory ID, and controller endpoints.
>
> These cannot be obtained through describeMetadataQuorum since that only 
> describes the existing quorum, not what you would like the new one to 
> be. Also, obviously, we never want to reuse an existing directory ID.
>
> Therefore I will vote -1 on this pending further discussion.
>
> best,
> Colin
>
> On Tue, Apr 22, 2025, at 08:56, Kuan Po Tseng wrote:
>> Hello everyone,
>>
>> I would like to start a vote on KIP-1141, thank you!
>>
>> KIP: https://cwiki.apache.org/confluence/x/PZFEF
>> Discussion thread:
>> https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4
>>
>> Best,
>> Kuan-Po Tseng

Re: [VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Posted by Colin McCabe <cm...@apache.org>.

Hi Kuan Po Tseng,

It is not possible to add a new controller purely by ID. You need controller ID, directory ID, and controller endpoints.

These cannot be obtained through describeMetadataQuorum since that only describes the existing quorum, not what you would like the new one to be. Also, obviously, we never want to reuse an existing directory ID.

Therefore I will vote -1 on this pending further discussion.

best,
Colin

On Tue, Apr 22, 2025, at 08:56, Kuan Po Tseng wrote:
> Hello everyone,
>
> I would like to start a vote on KIP-1141, thank you!
>
> KIP: https://cwiki.apache.org/confluence/x/PZFEF
> Discussion thread:
> https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4
>
> Best,
> Kuan-Po Tseng

Re: [VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Posted by Kuan Po Tseng <br...@apache.org>.

Hi all,

Thanks everyone for the votes.
This KIP is now accepted with +3 binding (Chia-Pin Tsai, Luke Chen, PoAn Yang) votes.

Best,
Kuan-Po

On 2026/05/12 01:03:41 Chia-Ping Tsai wrote:
> Yes, v4.4.0 is a good place to start this new feature!
> 
> Luke Chen <sh...@gmail.com> 於 2026年5月12日週二 上午8:27寫道：
> 
> > Hi all,
> >
> > Since kafka v4.3.0 is almost out and no other comments on this, I think we
> > can target this KIP in kafka v4.4.0.
> > WDYT?
> >
> > Thanks,
> > Luke
> >
> > On Mon, Mar 16, 2026 at 2:52 AM Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> >
> > > Although this vote has received three +1 (binding) votes, I think it
> > would
> > > be best to keep it open a bit longer for more eyes, especially given the
> > > previous -1. I'm hopping Colin might have some time to take another look
> > at
> > > the clarification. In the meantime, Kuan-Po will prepare a draft PR so we
> > > can dig into the implementation details.
> > >
> > > On 2025/04/22 15:56:53 Kuan Po Tseng wrote:
> > > > Hello everyone,
> > > >
> > > > I would like to start a vote on KIP-1141, thank you!
> > > >
> > > > KIP: https://cwiki.apache.org/confluence/x/PZFEF
> > > > Discussion thread:
> > > > https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4
> > > >
> > > > Best,
> > > > Kuan-Po Tseng
> > > >
> > >
> >
>

Re: [VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Posted by Chia-Ping Tsai <ch...@gmail.com>.

Yes, v4.4.0 is a good place to start this new feature!

Luke Chen <sh...@gmail.com> 於 2026年5月12日週二 上午8:27寫道：

> Hi all,
>
> Since kafka v4.3.0 is almost out and no other comments on this, I think we
> can target this KIP in kafka v4.4.0.
> WDYT?
>
> Thanks,
> Luke
>
> On Mon, Mar 16, 2026 at 2:52 AM Chia-Ping Tsai <ch...@apache.org>
> wrote:
>
> > Although this vote has received three +1 (binding) votes, I think it
> would
> > be best to keep it open a bit longer for more eyes, especially given the
> > previous -1. I'm hopping Colin might have some time to take another look
> at
> > the clarification. In the meantime, Kuan-Po will prepare a draft PR so we
> > can dig into the implementation details.
> >
> > On 2025/04/22 15:56:53 Kuan Po Tseng wrote:
> > > Hello everyone,
> > >
> > > I would like to start a vote on KIP-1141, thank you!
> > >
> > > KIP: https://cwiki.apache.org/confluence/x/PZFEF
> > > Discussion thread:
> > > https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4
> > >
> > > Best,
> > > Kuan-Po Tseng
> > >
> >
>

Re: [VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Posted by Luke Chen <sh...@gmail.com>.

Hi all,

Since kafka v4.3.0 is almost out and no other comments on this, I think we
can target this KIP in kafka v4.4.0.
WDYT?

Thanks,
Luke

On Mon, Mar 16, 2026 at 2:52 AM Chia-Ping Tsai <ch...@apache.org> wrote:

> Although this vote has received three +1 (binding) votes, I think it would
> be best to keep it open a bit longer for more eyes, especially given the
> previous -1. I'm hopping Colin might have some time to take another look at
> the clarification. In the meantime, Kuan-Po will prepare a draft PR so we
> can dig into the implementation details.
>
> On 2025/04/22 15:56:53 Kuan Po Tseng wrote:
> > Hello everyone,
> >
> > I would like to start a vote on KIP-1141, thank you!
> >
> > KIP: https://cwiki.apache.org/confluence/x/PZFEF
> > Discussion thread:
> > https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4
> >
> > Best,
> > Kuan-Po Tseng
> >
>

Re: [VOTE] KIP-1141: Simplifying MetadataQuorumCommand by Leveraging Admin API for Controller Management

Posted by Chia-Ping Tsai <ch...@apache.org>.

Although this vote has received three +1 (binding) votes, I think it would be best to keep it open a bit longer for more eyes, especially given the previous -1. I'm hopping Colin might have some time to take another look at the clarification. In the meantime, Kuan-Po will prepare a draft PR so we can dig into the implementation details.

On 2025/04/22 15:56:53 Kuan Po Tseng wrote:
> Hello everyone,
> 
> I would like to start a vote on KIP-1141, thank you!
> 
> KIP: https://cwiki.apache.org/confluence/x/PZFEF
> Discussion thread:
> https://lists.apache.org/thread/mq5n8j8w0ycg9fhd1m4zcc2ctrgfd0d4
> 
> Best,
> Kuan-Po Tseng
>