[DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hello all,

I would like to start a discussion on KIP-1262, which proposes removing the
formatting requirement for brokers and observer controllers. Currently, I
am considering two high-level designs, and would appreciate community
feedback on both approaches to decide on a final design.

KIP link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories

Best,
Kevin Wu

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi Alyssa,

RE AH1: Yeah, I will make the language more explicit.

RE AH2: I should have said "Prior to this KIP, starting kafka on any node
requires cluster.id to be present in meta.properties, otherwise the node
will crash. This means that clusters that upgrade to an MV that supports
this feature will have a cluster.id." Apologies for the confusion.

Best,
Kevin Wu

On Thu, Feb 26, 2026 at 7:38 PM Alyssa Huang via dev <de...@kafka.apache.org>
wrote:

> Hi Kevin,
>
> Thanks for the quick response!
>
> AH1: If the goal of the KIP is to remove the need to run `kafka-storage
> format` before running kafka on only a subset of the nodes in a cluster, I
> think it would be worth making that very clear (maybe at the beginning of
> Proposed Changes in a summary that details what the new format/startup
> procedure looks like).
>
> AH2:
>
> > Mainly because going forward,
> > meta.properties V2 will be written on newly formatted nodes +
> > non-formatting nodes. Additionally, starting kafka on any node requires
> > cluster.id to be present in meta.properties. Otherwise, the node
> crashes.
>
> The above seems confusing to me - are we saying that cluster.id will be
> required in meta.properties to start any kind of node? In your response to
> AH6 you seem to say the opposite?
>
> > Right now, starting kafka on any node requires cluster.id to be
> > present in meta.properties. Otherwise, the node crashes. As part of this
> > KIP, we no longer want that to be the case for every node.
>
>
> Best,
> Alyssa
>
> On Wed, Feb 25, 2026 at 2:23 PM Kevin Wu <ke...@gmail.com> wrote:
>
> > Hi Alyssa,
> >
> > Thanks for the replies and questions.
> >
> > RE AH1: The goal of the KIP (at least for now) is to remove the need to
> run
> > `kafka-storage format` before running kafka on brokers and observer
> > controllers. Sure, I can change the wording.
> >
> > RE AH2: I don't think you would need to? Mainly because going forward,
> > meta.properties V2 will be written on newly formatted nodes +
> > non-formatting nodes. Additionally, starting kafka on any node requires
> > cluster.id to be present in meta.properties. Otherwise, the node
> crashes.
> >
> > RE AH5: I specifically mentioned the `.checkpoint` file created from
> > formatting as part of this write path because the active controller
> writes
> > its contents to the log when it becomes leader. If there is no
> > bootstrap/0-0.checkpoint on the leader, it will default to the latest MV,
> > and after this KIP, that would mean the active controller would write
> > ClusterIdRecord alongside the MV during the bootstrap metadata write.
> >
> > RE AH6: Right now, starting kafka on any node requires cluster.id to be
> > present in meta.properties. Otherwise, the node crashes. As part of this
> > KIP, we no longer want that to be the case for every node. There are two
> > approaches for this, one that I think is more suitable for a minor
> release,
> > and one I think is better for 5.0 due to compatibility cases (although
> this
> > is definitely still up for discussion and I'd like to know everyone's
> > thoughts).
> > The former is what I described in the KIP, which is that any kafka node
> > that can become the active controller must have meta.properties with a
> > cluster.id to start kafka successfully. This removes the formatting
> > requirement for broker-only nodes and controllers who are not part of the
> > KRaft voter set. In this case, we do not need to cover for the case where
> > the leader does not have a meta.properties, because this node would have
> > crashed before becoming leader.
> > The latter is to remove the requirement for all nodes to format (keep in
> > mind that dynamic quorums currently must have at least one node format to
> > elect a leader in a proper configuration), but formatting would still be
> > optional to allow for control over feature levels etc.. In this case,
> kafka
> > would need to write its own cluster id upon electing an active
> controller.
> > However, we would then need to support dynamic quorum bootstrapping
> during
> > startup too. I don't think this is a good idea at the moment, as it would
> > mean introducing a static config to determine who are the "bootstrap
> > controllers," and the formatting logic for dynamic quorum is then
> > duplicated at startup. What happens if both the static config exist and
> the
> > node was formatted? You can imagine this same problem for every other
> > feature too. Basically, I think it is really confusing to users and bad
> > design if kafka can work without formatting any nodes, but also you can
> > format if you want. If we want to remove the requirement of any node
> having
> > to format, we should just remove the `kafka-storage format` and probably
> > the whole storage tool altogether, but that should be in a major release.
> > If we go with this approach now, another concern I have is about older MV
> > clusters whose nodes have software versions that support this feature.
> How
> > would a new software version broker persist the cluster id if the
> operator
> > did not format the node? I think it is easier to communicate how the
> latter
> > approach works in a major release.
> >
> > Best,
> > Kevin Wu
> >
> > On Wed, Feb 25, 2026 at 9:57 PM Alyssa Huang via dev <
> dev@kafka.apache.org
> > >
> > wrote:
> >
> > > Hey Kevin, thanks for the KIP!
> > >
> > > AH1:
> > >
> > > > The reason for this KIP is to remove the requirement of brokers
> needed
> > to
> > > > run the storage tool before starting Kafka.
> > >
> > > I'd like to clarify whether you're saying the goal of the KIP is to
> > remove
> > > the need of running just the storage format command or the storage tool
> > at
> > > all. Also, I misunderstood
> > >
> > > > Persisting this data does not need to be done before starting kafka.
> > >
> > >  as saying we don't need to persist this data at all and that actually
> > > messed up my understanding of the rest of your KIP on my first read -
> can
> > > we change the wording to "Persisting this data does not have to be done
> > > during storage formatting and can be done later during startup".
> > >
> > > AH2: In the MV upgrade case, do we see any value in updating the
> > > meta.properties file to v2?
> > >
> > > AH3:
> > >
> > > > Since this feature is associated with a new metadata record and
> > > > MetadataVersion, broker bootstrapping with cluster ID is required on
> > all
> > > > MVs < X where X is the first MV that supports this feature. Because
> > some
> > > > MetadataVersion is resolved during each node's formatting, we can
> > > determine
> > > > at format time if a ClusterIdRecord is needed as part of a
> controller's
> > > > 0-0/bootstrap.checkpoint.
> > >
> > > Can you go into some detail on how the MV is resolved during a node's
> > > formatting? This might be interesting for readers to know in the
> > > provisioning and re-formatting/adding a single node case.
> > >
> > > AH4: Under Proposed Changes the first header is:
> > >
> > > > meta.properties can be written during kafka broker/controller startup
> > if
> > > > it doesn't exist already (from formatting)
> > >
> > > Nit, but should this be 'meta.properties' *will* be written during
> kafka
> > > broker/controller startup if it doesn't exist already?
> > >
> > > AH5: Could the following be expanded a bit? "write-path" might be too
> > > generic since all nodes will write the MV they are formatted with to
> the
> > > bootstrap metadata (but only on leader election will the elected node
> > > attempt to write the bootstrap metadata MV to the metadata log). I
> found
> > > "formatting a node (specifically controllers who can become leader)"
> > > especially confusing, since I'm assuming you're saying the
> > > write-to-metadata-log path for MV.
> > >
> > > > This is enforceable along the write-path for MV, which occurs at two
> > > > points: formatting a node (specifically controllers who can become
> > > leader)
> > > > and upgrading the MV using kafka-features upgrade
> > >
> > >
> > > AH6:
> > >
> > > > However, kafka should still be able to handle the case where a leader
> > is
> > > > elected who does not have clusterId in meta.properties, which can
> occur
> > > if
> > > > a majority of voters do not have a clusterId
> > >
> > > The wording makes it sound like this is the exceptional case, but
> before
> > I
> > > read JR6 and your respond I had assumed this would be the normal case
> > for a
> > > cluster provisioned with KIP-1262 support. Can we make this more clear
> in
> > > the KIP? Why are we making cluster id optional in meta.properties v2 if
> > we
> > > expect to still write it?
> > >
> > > We can still enforce that bootstrap controllers must have formatted
> (and
> > > > therefore persisted a cluster id to meta.properties) prior to
> starting
> > > > kafka.
> > >
> > > I'm confused if this is saying we will continue to enforce persisting
> > > cluster id to meta.properties (given that you've also said kafka will
> > > handle the case where a leader is elected w/o clusterid in its
> > > meta.properties)
> > >
> > > AH7:
> > >
> > > > Nodes discover persist cluster id to meta.properties from metadata
> > > > publishing pipeline
> > > >
> > > Seems like there's an extra word in this header
> > >
> > > Thanks in advance for your responses :)
> > > Alyssa
> > >
> > > On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <ke...@gmail.com>
> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thanks for the replies and questions.
> > > >
> > > > RE JR1: Updated the KIP with the record schema for ClusterIdRecord.
> One
> > > > thing I'm not sure about yet is whether or not the record field
> should
> > be
> > > > of UUID or String type. This is because kafka's quickstart docs refer
> > to
> > > > setting `--cluster-id` to a UUID in the storage tool. However, many
> > > places
> > > > in kafka broker/controller code (e.g. the raft client, broker
> lifecycle
> > > > manager, and even the formatter itself) only require this type to be
> a
> > > > String. Since not all Strings are valid UUIDs, making this record
> field
> > > of
> > > > type UUID might be too restrictive and complicate upgrading the MV
> for
> > > > existing clusters, since they might have a non-UUID cluster id
> string,
> > > but
> > > > need to write this record when upgrading to an MV that supports this
> > > > feature. Let me know what you think.
> > > >
> > > > RE JR2: Any controller node formatted with `--standalone,
> > > > --initial-controllers` or who is part of the static voter set defined
> > by
> > > > `controller.quorum.voters` can write the ClusterIdRecord by including
> > the
> > > > `--cluster-id` argument to `kafka-storage format`. However, if the MV
> > of
> > > > the cluster supports it, there is exactly one writer of this record
> to
> > > the
> > > > cluster metadata partition. The writer is the first active
> controller,
> > > who
> > > > writes this record alongside other bootstrap metadata records (e.g.
> > > > metadata version) during controller activation. At this point, we
> > already
> > > > depend on MV existing, since the active controller writes these
> > bootstrap
> > > > metadata records as a transaction if the MV supports it. I think
> > writing
> > > > the cluster id record would follow a similar pattern.
> > > >
> > > > RE JR3: When a node formats, it will write the meta.properties file.
> > > During
> > > > formatting, a node must resolve the MV it wants to format with, which
> > is
> > > > explained more in RE JR5. I need to think about this more, but I
> think
> > we
> > > > should keep `--cluster-id` as a required flag for invoking the format
> > > > command. If a broker/observer controller does not format,
> > meta.properties
> > > > is written without cluster id immediately after startup (i.e. where
> we
> > > read
> > > > it from disk now in KafkaRaftServer).
> > > >
> > > > RE JR4: Yeah, will do. In this context, when I say observers I'm
> > > referring
> > > > to any controllers who are not part of the KRaft voter set when they
> > > start
> > > > kafka, or any brokers. I will make this explicit in the KIP. From the
> > > > perspective of this feature and KRaft leader election, controller
> nodes
> > > who
> > > > format with `--no-initial-controllers`, controller nodes who are not
> > part
> > > > of `controller.quorum.voters`, and brokers, all do not "need" to
> > format,
> > > > since they cannot become the active controller. This means they can
> > > resolve
> > > > metadata like the cluster id after discovering the leader. We have a
> > > > similar pattern with how controller nodes who format with
> > > > `--no-initial-controllers` discover the kraft version of the cluster.
> > > >
> > > > RE JR5: If a node formats, it must resolve a metadata version with
> > which
> > > to
> > > > format. This comes from the `--release-version/--feature` flag and
> > > defaults
> > > > to the latest production MV. Therefore, when a node formats with a
> > > metadata
> > > > version that supports this feature, it will write the ClusterIdRecord
> > to
> > > > its `0-0/bootstrap.checkpoint`. If the node formats with a metadata
> > > version
> > > > that does not support this feature, it does not write ClusterIdRecord
> > to
> > > > its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is
> > assumed
> > > > that this node is part of a cluster whose MV supports this.
> Otherwise,
> > > this
> > > > is a misconfiguration and the node will fail to register with the
> > leader
> > > > since there is no way for it to persist cluster id to its
> > meta.properties
> > > > without formatting.
> > > >
> > > > Although I did not specify this yet on the KIP explicitly, after some
> > > > offline discussion I think it makes sense to enforce the following
> > > > invariant as part of the feature design: if the persisted metadata
> > > version
> > > > supports this feature, the ClusterId record must also be persisted.
> > This
> > > is
> > > > enforceable on the write-path for MV, which occurs at two points--
> > during
> > > > formatting and during feature upgrades. There is a similar pattern
> with
> > > > kraft.version, as it gets written to disk at the same two points.
> > > >
> > > > RE JR6: The main motivation for writing cluster id to meta.properties
> > as
> > > > well is because it can act as a projection of the cluster metadata
> > > > partition which essentially only exposes the cluster id to readers.
> For
> > > > example, the raft layer needs to be aware of the cluster id for its
> own
> > > RPC
> > > > handling/validation, but raft cannot read metadata records. There are
> > > many
> > > > readers of this cluster id value during the startup of the cluster.
> > > > Therefore, avoiding a read of the metadata partition to discover the
> > > value
> > > > of this metadata will prevent more complications of the startup code.
> > > >
> > > > Best,
> > > > Kevin Wu
> > > >
> > > >
> > > > On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <
> dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hi, Kevin,
> > > > >
> > > > > Thanks for the KIP. A few comments.
> > > > >
> > > > > JR1. ClusterIdRecord : Could you define the record format?
> > > > >
> > > > > JR2. "a new MetadataVersion that supports encoding/decoding this
> > > record.
> > > > > This means that during formatting, the bootstrap ClusterIdRecord is
> > > only
> > > > > written if the cluster is formatted with a MV that supports this
> > > > feature."
> > > > > Could you describe who writes the ClusterIdRecord? Is it the leader
> > > > > controller? Also, when is the record written? Do we guarantee that
> MV
> > > is
> > > > > available at that time?
> > > > >
> > > > > JR3. "meta.properties can be written during kafka broker/controller
> > > > startup
> > > > > if it doesn't exist already (from formatting)"
> > > > > Could you describe when meta.properties is written? Is MV available
> > at
> > > > that
> > > > > time?
> > > > >
> > > > > JR4. "Introduce a metadata record for cluster id + observers
> persist
> > > > > cluster id to meta.properties from metadata publishing pipeline"
> > > > > Could you clarify what observers are? Are they observer controllers
> > or
> > > > are
> > > > > they brokers (which are referred to as observers to the
> controller)?
> > > > >
> > > > > JR5. "Bootstrap controllers can add a mandatory “cluster id” record
> > > > during
> > > > > formatting"
> > > > > This sounds like adding a ClusterIdRecord is optional. If so, could
> > you
> > > > > describe when a record will be added and when a record will not be
> > > added?
> > > > >
> > > > > JR6. "However, kafka should still be able to handle the case where
> a
> > > > leader
> > > > > is elected without a cluster id in meta.properties , since KRaft
> does
> > > not
> > > > > need cluster.id  in order to elect a leader.
> > > > >           In this case, the active controller will write a cluster
> id
> > > > > record during the bootstrap metadata write."
> > > > > Hmm, earlier, the KIP says "Upon discovering the cluster ID for the
> > > first
> > > > > time, these nodes need to persist this to meta.properties". Why do
> we
> > > > need
> > > > > to introduce a separate place to write the cluster id to
> > > > meta.properties.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <ke...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Manually bumping this thread after finalizing a design.
> > > > > > KIP link:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > > >
> > > > > > Best,
> > > > > > Kevin Wu
> > > > > >
> > > > > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hello all,
> > > > > > >
> > > > > > > I would like to start a discussion on KIP-1262, which proposes
> > > > removing
> > > > > > > the formatting requirement for brokers and observer
> controllers.
> > > > > > Currently,
> > > > > > > I am considering two high-level designs, and would appreciate
> > > > community
> > > > > > > feedback on both approaches to decide on a final design.
> > > > > > >
> > > > > > > KIP link:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > > > >
> > > > > > > Best,
> > > > > > > Kevin Wu
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Alyssa Huang via dev <de...@kafka.apache.org>]

Hi Kevin,

Thanks for the quick response!

AH1: If the goal of the KIP is to remove the need to run `kafka-storage
format` before running kafka on only a subset of the nodes in a cluster, I
think it would be worth making that very clear (maybe at the beginning of
Proposed Changes in a summary that details what the new format/startup
procedure looks like).

AH2:

> Mainly because going forward,
> meta.properties V2 will be written on newly formatted nodes +
> non-formatting nodes. Additionally, starting kafka on any node requires
> cluster.id to be present in meta.properties. Otherwise, the node crashes.

The above seems confusing to me - are we saying that cluster.id will be
required in meta.properties to start any kind of node? In your response to
AH6 you seem to say the opposite?

> Right now, starting kafka on any node requires cluster.id to be
> present in meta.properties. Otherwise, the node crashes. As part of this
> KIP, we no longer want that to be the case for every node.


Best,
Alyssa

On Wed, Feb 25, 2026 at 2:23 PM Kevin Wu <ke...@gmail.com> wrote:

> Hi Alyssa,
>
> Thanks for the replies and questions.
>
> RE AH1: The goal of the KIP (at least for now) is to remove the need to run
> `kafka-storage format` before running kafka on brokers and observer
> controllers. Sure, I can change the wording.
>
> RE AH2: I don't think you would need to? Mainly because going forward,
> meta.properties V2 will be written on newly formatted nodes +
> non-formatting nodes. Additionally, starting kafka on any node requires
> cluster.id to be present in meta.properties. Otherwise, the node crashes.
>
> RE AH5: I specifically mentioned the `.checkpoint` file created from
> formatting as part of this write path because the active controller writes
> its contents to the log when it becomes leader. If there is no
> bootstrap/0-0.checkpoint on the leader, it will default to the latest MV,
> and after this KIP, that would mean the active controller would write
> ClusterIdRecord alongside the MV during the bootstrap metadata write.
>
> RE AH6: Right now, starting kafka on any node requires cluster.id to be
> present in meta.properties. Otherwise, the node crashes. As part of this
> KIP, we no longer want that to be the case for every node. There are two
> approaches for this, one that I think is more suitable for a minor release,
> and one I think is better for 5.0 due to compatibility cases (although this
> is definitely still up for discussion and I'd like to know everyone's
> thoughts).
> The former is what I described in the KIP, which is that any kafka node
> that can become the active controller must have meta.properties with a
> cluster.id to start kafka successfully. This removes the formatting
> requirement for broker-only nodes and controllers who are not part of the
> KRaft voter set. In this case, we do not need to cover for the case where
> the leader does not have a meta.properties, because this node would have
> crashed before becoming leader.
> The latter is to remove the requirement for all nodes to format (keep in
> mind that dynamic quorums currently must have at least one node format to
> elect a leader in a proper configuration), but formatting would still be
> optional to allow for control over feature levels etc.. In this case, kafka
> would need to write its own cluster id upon electing an active controller.
> However, we would then need to support dynamic quorum bootstrapping during
> startup too. I don't think this is a good idea at the moment, as it would
> mean introducing a static config to determine who are the "bootstrap
> controllers," and the formatting logic for dynamic quorum is then
> duplicated at startup. What happens if both the static config exist and the
> node was formatted? You can imagine this same problem for every other
> feature too. Basically, I think it is really confusing to users and bad
> design if kafka can work without formatting any nodes, but also you can
> format if you want. If we want to remove the requirement of any node having
> to format, we should just remove the `kafka-storage format` and probably
> the whole storage tool altogether, but that should be in a major release.
> If we go with this approach now, another concern I have is about older MV
> clusters whose nodes have software versions that support this feature. How
> would a new software version broker persist the cluster id if the operator
> did not format the node? I think it is easier to communicate how the latter
> approach works in a major release.
>
> Best,
> Kevin Wu
>
> On Wed, Feb 25, 2026 at 9:57 PM Alyssa Huang via dev <dev@kafka.apache.org
> >
> wrote:
>
> > Hey Kevin, thanks for the KIP!
> >
> > AH1:
> >
> > > The reason for this KIP is to remove the requirement of brokers needed
> to
> > > run the storage tool before starting Kafka.
> >
> > I'd like to clarify whether you're saying the goal of the KIP is to
> remove
> > the need of running just the storage format command or the storage tool
> at
> > all. Also, I misunderstood
> >
> > > Persisting this data does not need to be done before starting kafka.
> >
> >  as saying we don't need to persist this data at all and that actually
> > messed up my understanding of the rest of your KIP on my first read - can
> > we change the wording to "Persisting this data does not have to be done
> > during storage formatting and can be done later during startup".
> >
> > AH2: In the MV upgrade case, do we see any value in updating the
> > meta.properties file to v2?
> >
> > AH3:
> >
> > > Since this feature is associated with a new metadata record and
> > > MetadataVersion, broker bootstrapping with cluster ID is required on
> all
> > > MVs < X where X is the first MV that supports this feature. Because
> some
> > > MetadataVersion is resolved during each node's formatting, we can
> > determine
> > > at format time if a ClusterIdRecord is needed as part of a controller's
> > > 0-0/bootstrap.checkpoint.
> >
> > Can you go into some detail on how the MV is resolved during a node's
> > formatting? This might be interesting for readers to know in the
> > provisioning and re-formatting/adding a single node case.
> >
> > AH4: Under Proposed Changes the first header is:
> >
> > > meta.properties can be written during kafka broker/controller startup
> if
> > > it doesn't exist already (from formatting)
> >
> > Nit, but should this be 'meta.properties' *will* be written during kafka
> > broker/controller startup if it doesn't exist already?
> >
> > AH5: Could the following be expanded a bit? "write-path" might be too
> > generic since all nodes will write the MV they are formatted with to the
> > bootstrap metadata (but only on leader election will the elected node
> > attempt to write the bootstrap metadata MV to the metadata log). I found
> > "formatting a node (specifically controllers who can become leader)"
> > especially confusing, since I'm assuming you're saying the
> > write-to-metadata-log path for MV.
> >
> > > This is enforceable along the write-path for MV, which occurs at two
> > > points: formatting a node (specifically controllers who can become
> > leader)
> > > and upgrading the MV using kafka-features upgrade
> >
> >
> > AH6:
> >
> > > However, kafka should still be able to handle the case where a leader
> is
> > > elected who does not have clusterId in meta.properties, which can occur
> > if
> > > a majority of voters do not have a clusterId
> >
> > The wording makes it sound like this is the exceptional case, but before
> I
> > read JR6 and your respond I had assumed this would be the normal case
> for a
> > cluster provisioned with KIP-1262 support. Can we make this more clear in
> > the KIP? Why are we making cluster id optional in meta.properties v2 if
> we
> > expect to still write it?
> >
> > We can still enforce that bootstrap controllers must have formatted (and
> > > therefore persisted a cluster id to meta.properties) prior to starting
> > > kafka.
> >
> > I'm confused if this is saying we will continue to enforce persisting
> > cluster id to meta.properties (given that you've also said kafka will
> > handle the case where a leader is elected w/o clusterid in its
> > meta.properties)
> >
> > AH7:
> >
> > > Nodes discover persist cluster id to meta.properties from metadata
> > > publishing pipeline
> > >
> > Seems like there's an extra word in this header
> >
> > Thanks in advance for your responses :)
> > Alyssa
> >
> > On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <ke...@gmail.com> wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for the replies and questions.
> > >
> > > RE JR1: Updated the KIP with the record schema for ClusterIdRecord. One
> > > thing I'm not sure about yet is whether or not the record field should
> be
> > > of UUID or String type. This is because kafka's quickstart docs refer
> to
> > > setting `--cluster-id` to a UUID in the storage tool. However, many
> > places
> > > in kafka broker/controller code (e.g. the raft client, broker lifecycle
> > > manager, and even the formatter itself) only require this type to be a
> > > String. Since not all Strings are valid UUIDs, making this record field
> > of
> > > type UUID might be too restrictive and complicate upgrading the MV for
> > > existing clusters, since they might have a non-UUID cluster id string,
> > but
> > > need to write this record when upgrading to an MV that supports this
> > > feature. Let me know what you think.
> > >
> > > RE JR2: Any controller node formatted with `--standalone,
> > > --initial-controllers` or who is part of the static voter set defined
> by
> > > `controller.quorum.voters` can write the ClusterIdRecord by including
> the
> > > `--cluster-id` argument to `kafka-storage format`. However, if the MV
> of
> > > the cluster supports it, there is exactly one writer of this record to
> > the
> > > cluster metadata partition. The writer is the first active controller,
> > who
> > > writes this record alongside other bootstrap metadata records (e.g.
> > > metadata version) during controller activation. At this point, we
> already
> > > depend on MV existing, since the active controller writes these
> bootstrap
> > > metadata records as a transaction if the MV supports it. I think
> writing
> > > the cluster id record would follow a similar pattern.
> > >
> > > RE JR3: When a node formats, it will write the meta.properties file.
> > During
> > > formatting, a node must resolve the MV it wants to format with, which
> is
> > > explained more in RE JR5. I need to think about this more, but I think
> we
> > > should keep `--cluster-id` as a required flag for invoking the format
> > > command. If a broker/observer controller does not format,
> meta.properties
> > > is written without cluster id immediately after startup (i.e. where we
> > read
> > > it from disk now in KafkaRaftServer).
> > >
> > > RE JR4: Yeah, will do. In this context, when I say observers I'm
> > referring
> > > to any controllers who are not part of the KRaft voter set when they
> > start
> > > kafka, or any brokers. I will make this explicit in the KIP. From the
> > > perspective of this feature and KRaft leader election, controller nodes
> > who
> > > format with `--no-initial-controllers`, controller nodes who are not
> part
> > > of `controller.quorum.voters`, and brokers, all do not "need" to
> format,
> > > since they cannot become the active controller. This means they can
> > resolve
> > > metadata like the cluster id after discovering the leader. We have a
> > > similar pattern with how controller nodes who format with
> > > `--no-initial-controllers` discover the kraft version of the cluster.
> > >
> > > RE JR5: If a node formats, it must resolve a metadata version with
> which
> > to
> > > format. This comes from the `--release-version/--feature` flag and
> > defaults
> > > to the latest production MV. Therefore, when a node formats with a
> > metadata
> > > version that supports this feature, it will write the ClusterIdRecord
> to
> > > its `0-0/bootstrap.checkpoint`. If the node formats with a metadata
> > version
> > > that does not support this feature, it does not write ClusterIdRecord
> to
> > > its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is
> assumed
> > > that this node is part of a cluster whose MV supports this. Otherwise,
> > this
> > > is a misconfiguration and the node will fail to register with the
> leader
> > > since there is no way for it to persist cluster id to its
> meta.properties
> > > without formatting.
> > >
> > > Although I did not specify this yet on the KIP explicitly, after some
> > > offline discussion I think it makes sense to enforce the following
> > > invariant as part of the feature design: if the persisted metadata
> > version
> > > supports this feature, the ClusterId record must also be persisted.
> This
> > is
> > > enforceable on the write-path for MV, which occurs at two points--
> during
> > > formatting and during feature upgrades. There is a similar pattern with
> > > kraft.version, as it gets written to disk at the same two points.
> > >
> > > RE JR6: The main motivation for writing cluster id to meta.properties
> as
> > > well is because it can act as a projection of the cluster metadata
> > > partition which essentially only exposes the cluster id to readers. For
> > > example, the raft layer needs to be aware of the cluster id for its own
> > RPC
> > > handling/validation, but raft cannot read metadata records. There are
> > many
> > > readers of this cluster id value during the startup of the cluster.
> > > Therefore, avoiding a read of the metadata partition to discover the
> > value
> > > of this metadata will prevent more complications of the startup code.
> > >
> > > Best,
> > > Kevin Wu
> > >
> > >
> > > On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <de...@kafka.apache.org>
> > > wrote:
> > >
> > > > Hi, Kevin,
> > > >
> > > > Thanks for the KIP. A few comments.
> > > >
> > > > JR1. ClusterIdRecord : Could you define the record format?
> > > >
> > > > JR2. "a new MetadataVersion that supports encoding/decoding this
> > record.
> > > > This means that during formatting, the bootstrap ClusterIdRecord is
> > only
> > > > written if the cluster is formatted with a MV that supports this
> > > feature."
> > > > Could you describe who writes the ClusterIdRecord? Is it the leader
> > > > controller? Also, when is the record written? Do we guarantee that MV
> > is
> > > > available at that time?
> > > >
> > > > JR3. "meta.properties can be written during kafka broker/controller
> > > startup
> > > > if it doesn't exist already (from formatting)"
> > > > Could you describe when meta.properties is written? Is MV available
> at
> > > that
> > > > time?
> > > >
> > > > JR4. "Introduce a metadata record for cluster id + observers persist
> > > > cluster id to meta.properties from metadata publishing pipeline"
> > > > Could you clarify what observers are? Are they observer controllers
> or
> > > are
> > > > they brokers (which are referred to as observers to the controller)?
> > > >
> > > > JR5. "Bootstrap controllers can add a mandatory “cluster id” record
> > > during
> > > > formatting"
> > > > This sounds like adding a ClusterIdRecord is optional. If so, could
> you
> > > > describe when a record will be added and when a record will not be
> > added?
> > > >
> > > > JR6. "However, kafka should still be able to handle the case where a
> > > leader
> > > > is elected without a cluster id in meta.properties , since KRaft does
> > not
> > > > need cluster.id  in order to elect a leader.
> > > >           In this case, the active controller will write a cluster id
> > > > record during the bootstrap metadata write."
> > > > Hmm, earlier, the KIP says "Upon discovering the cluster ID for the
> > first
> > > > time, these nodes need to persist this to meta.properties". Why do we
> > > need
> > > > to introduce a separate place to write the cluster id to
> > > meta.properties.
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <ke...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Manually bumping this thread after finalizing a design.
> > > > > KIP link:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > >
> > > > > Best,
> > > > > Kevin Wu
> > > > >
> > > > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > I would like to start a discussion on KIP-1262, which proposes
> > > removing
> > > > > > the formatting requirement for brokers and observer controllers.
> > > > > Currently,
> > > > > > I am considering two high-level designs, and would appreciate
> > > community
> > > > > > feedback on both approaches to decide on a final design.
> > > > > >
> > > > > > KIP link:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > > >
> > > > > > Best,
> > > > > > Kevin Wu
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi Alyssa,

Thanks for the replies and questions.

RE AH1: The goal of the KIP (at least for now) is to remove the need to run
`kafka-storage format` before running kafka on brokers and observer
controllers. Sure, I can change the wording.

RE AH2: I don't think you would need to? Mainly because going forward,
meta.properties V2 will be written on newly formatted nodes +
non-formatting nodes. Additionally, starting kafka on any node requires
cluster.id to be present in meta.properties. Otherwise, the node crashes.

RE AH5: I specifically mentioned the `.checkpoint` file created from
formatting as part of this write path because the active controller writes
its contents to the log when it becomes leader. If there is no
bootstrap/0-0.checkpoint on the leader, it will default to the latest MV,
and after this KIP, that would mean the active controller would write
ClusterIdRecord alongside the MV during the bootstrap metadata write.

RE AH6: Right now, starting kafka on any node requires cluster.id to be
present in meta.properties. Otherwise, the node crashes. As part of this
KIP, we no longer want that to be the case for every node. There are two
approaches for this, one that I think is more suitable for a minor release,
and one I think is better for 5.0 due to compatibility cases (although this
is definitely still up for discussion and I'd like to know everyone's
thoughts).
The former is what I described in the KIP, which is that any kafka node
that can become the active controller must have meta.properties with a
cluster.id to start kafka successfully. This removes the formatting
requirement for broker-only nodes and controllers who are not part of the
KRaft voter set. In this case, we do not need to cover for the case where
the leader does not have a meta.properties, because this node would have
crashed before becoming leader.
The latter is to remove the requirement for all nodes to format (keep in
mind that dynamic quorums currently must have at least one node format to
elect a leader in a proper configuration), but formatting would still be
optional to allow for control over feature levels etc.. In this case, kafka
would need to write its own cluster id upon electing an active controller.
However, we would then need to support dynamic quorum bootstrapping during
startup too. I don't think this is a good idea at the moment, as it would
mean introducing a static config to determine who are the "bootstrap
controllers," and the formatting logic for dynamic quorum is then
duplicated at startup. What happens if both the static config exist and the
node was formatted? You can imagine this same problem for every other
feature too. Basically, I think it is really confusing to users and bad
design if kafka can work without formatting any nodes, but also you can
format if you want. If we want to remove the requirement of any node having
to format, we should just remove the `kafka-storage format` and probably
the whole storage tool altogether, but that should be in a major release.
If we go with this approach now, another concern I have is about older MV
clusters whose nodes have software versions that support this feature. How
would a new software version broker persist the cluster id if the operator
did not format the node? I think it is easier to communicate how the latter
approach works in a major release.

Best,
Kevin Wu

On Wed, Feb 25, 2026 at 9:57 PM Alyssa Huang via dev <de...@kafka.apache.org>
wrote:

> Hey Kevin, thanks for the KIP!
>
> AH1:
>
> > The reason for this KIP is to remove the requirement of brokers needed to
> > run the storage tool before starting Kafka.
>
> I'd like to clarify whether you're saying the goal of the KIP is to remove
> the need of running just the storage format command or the storage tool at
> all. Also, I misunderstood
>
> > Persisting this data does not need to be done before starting kafka.
>
>  as saying we don't need to persist this data at all and that actually
> messed up my understanding of the rest of your KIP on my first read - can
> we change the wording to "Persisting this data does not have to be done
> during storage formatting and can be done later during startup".
>
> AH2: In the MV upgrade case, do we see any value in updating the
> meta.properties file to v2?
>
> AH3:
>
> > Since this feature is associated with a new metadata record and
> > MetadataVersion, broker bootstrapping with cluster ID is required on all
> > MVs < X where X is the first MV that supports this feature. Because some
> > MetadataVersion is resolved during each node's formatting, we can
> determine
> > at format time if a ClusterIdRecord is needed as part of a controller's
> > 0-0/bootstrap.checkpoint.
>
> Can you go into some detail on how the MV is resolved during a node's
> formatting? This might be interesting for readers to know in the
> provisioning and re-formatting/adding a single node case.
>
> AH4: Under Proposed Changes the first header is:
>
> > meta.properties can be written during kafka broker/controller startup if
> > it doesn't exist already (from formatting)
>
> Nit, but should this be 'meta.properties' *will* be written during kafka
> broker/controller startup if it doesn't exist already?
>
> AH5: Could the following be expanded a bit? "write-path" might be too
> generic since all nodes will write the MV they are formatted with to the
> bootstrap metadata (but only on leader election will the elected node
> attempt to write the bootstrap metadata MV to the metadata log). I found
> "formatting a node (specifically controllers who can become leader)"
> especially confusing, since I'm assuming you're saying the
> write-to-metadata-log path for MV.
>
> > This is enforceable along the write-path for MV, which occurs at two
> > points: formatting a node (specifically controllers who can become
> leader)
> > and upgrading the MV using kafka-features upgrade
>
>
> AH6:
>
> > However, kafka should still be able to handle the case where a leader is
> > elected who does not have clusterId in meta.properties, which can occur
> if
> > a majority of voters do not have a clusterId
>
> The wording makes it sound like this is the exceptional case, but before I
> read JR6 and your respond I had assumed this would be the normal case for a
> cluster provisioned with KIP-1262 support. Can we make this more clear in
> the KIP? Why are we making cluster id optional in meta.properties v2 if we
> expect to still write it?
>
> We can still enforce that bootstrap controllers must have formatted (and
> > therefore persisted a cluster id to meta.properties) prior to starting
> > kafka.
>
> I'm confused if this is saying we will continue to enforce persisting
> cluster id to meta.properties (given that you've also said kafka will
> handle the case where a leader is elected w/o clusterid in its
> meta.properties)
>
> AH7:
>
> > Nodes discover persist cluster id to meta.properties from metadata
> > publishing pipeline
> >
> Seems like there's an extra word in this header
>
> Thanks in advance for your responses :)
> Alyssa
>
> On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <ke...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the replies and questions.
> >
> > RE JR1: Updated the KIP with the record schema for ClusterIdRecord. One
> > thing I'm not sure about yet is whether or not the record field should be
> > of UUID or String type. This is because kafka's quickstart docs refer to
> > setting `--cluster-id` to a UUID in the storage tool. However, many
> places
> > in kafka broker/controller code (e.g. the raft client, broker lifecycle
> > manager, and even the formatter itself) only require this type to be a
> > String. Since not all Strings are valid UUIDs, making this record field
> of
> > type UUID might be too restrictive and complicate upgrading the MV for
> > existing clusters, since they might have a non-UUID cluster id string,
> but
> > need to write this record when upgrading to an MV that supports this
> > feature. Let me know what you think.
> >
> > RE JR2: Any controller node formatted with `--standalone,
> > --initial-controllers` or who is part of the static voter set defined by
> > `controller.quorum.voters` can write the ClusterIdRecord by including the
> > `--cluster-id` argument to `kafka-storage format`. However, if the MV of
> > the cluster supports it, there is exactly one writer of this record to
> the
> > cluster metadata partition. The writer is the first active controller,
> who
> > writes this record alongside other bootstrap metadata records (e.g.
> > metadata version) during controller activation. At this point, we already
> > depend on MV existing, since the active controller writes these bootstrap
> > metadata records as a transaction if the MV supports it. I think writing
> > the cluster id record would follow a similar pattern.
> >
> > RE JR3: When a node formats, it will write the meta.properties file.
> During
> > formatting, a node must resolve the MV it wants to format with, which is
> > explained more in RE JR5. I need to think about this more, but I think we
> > should keep `--cluster-id` as a required flag for invoking the format
> > command. If a broker/observer controller does not format, meta.properties
> > is written without cluster id immediately after startup (i.e. where we
> read
> > it from disk now in KafkaRaftServer).
> >
> > RE JR4: Yeah, will do. In this context, when I say observers I'm
> referring
> > to any controllers who are not part of the KRaft voter set when they
> start
> > kafka, or any brokers. I will make this explicit in the KIP. From the
> > perspective of this feature and KRaft leader election, controller nodes
> who
> > format with `--no-initial-controllers`, controller nodes who are not part
> > of `controller.quorum.voters`, and brokers, all do not "need" to format,
> > since they cannot become the active controller. This means they can
> resolve
> > metadata like the cluster id after discovering the leader. We have a
> > similar pattern with how controller nodes who format with
> > `--no-initial-controllers` discover the kraft version of the cluster.
> >
> > RE JR5: If a node formats, it must resolve a metadata version with which
> to
> > format. This comes from the `--release-version/--feature` flag and
> defaults
> > to the latest production MV. Therefore, when a node formats with a
> metadata
> > version that supports this feature, it will write the ClusterIdRecord to
> > its `0-0/bootstrap.checkpoint`. If the node formats with a metadata
> version
> > that does not support this feature, it does not write ClusterIdRecord to
> > its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is assumed
> > that this node is part of a cluster whose MV supports this. Otherwise,
> this
> > is a misconfiguration and the node will fail to register with the leader
> > since there is no way for it to persist cluster id to its meta.properties
> > without formatting.
> >
> > Although I did not specify this yet on the KIP explicitly, after some
> > offline discussion I think it makes sense to enforce the following
> > invariant as part of the feature design: if the persisted metadata
> version
> > supports this feature, the ClusterId record must also be persisted. This
> is
> > enforceable on the write-path for MV, which occurs at two points-- during
> > formatting and during feature upgrades. There is a similar pattern with
> > kraft.version, as it gets written to disk at the same two points.
> >
> > RE JR6: The main motivation for writing cluster id to meta.properties as
> > well is because it can act as a projection of the cluster metadata
> > partition which essentially only exposes the cluster id to readers. For
> > example, the raft layer needs to be aware of the cluster id for its own
> RPC
> > handling/validation, but raft cannot read metadata records. There are
> many
> > readers of this cluster id value during the startup of the cluster.
> > Therefore, avoiding a read of the metadata partition to discover the
> value
> > of this metadata will prevent more complications of the startup code.
> >
> > Best,
> > Kevin Wu
> >
> >
> > On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> > > Hi, Kevin,
> > >
> > > Thanks for the KIP. A few comments.
> > >
> > > JR1. ClusterIdRecord : Could you define the record format?
> > >
> > > JR2. "a new MetadataVersion that supports encoding/decoding this
> record.
> > > This means that during formatting, the bootstrap ClusterIdRecord is
> only
> > > written if the cluster is formatted with a MV that supports this
> > feature."
> > > Could you describe who writes the ClusterIdRecord? Is it the leader
> > > controller? Also, when is the record written? Do we guarantee that MV
> is
> > > available at that time?
> > >
> > > JR3. "meta.properties can be written during kafka broker/controller
> > startup
> > > if it doesn't exist already (from formatting)"
> > > Could you describe when meta.properties is written? Is MV available at
> > that
> > > time?
> > >
> > > JR4. "Introduce a metadata record for cluster id + observers persist
> > > cluster id to meta.properties from metadata publishing pipeline"
> > > Could you clarify what observers are? Are they observer controllers or
> > are
> > > they brokers (which are referred to as observers to the controller)?
> > >
> > > JR5. "Bootstrap controllers can add a mandatory “cluster id” record
> > during
> > > formatting"
> > > This sounds like adding a ClusterIdRecord is optional. If so, could you
> > > describe when a record will be added and when a record will not be
> added?
> > >
> > > JR6. "However, kafka should still be able to handle the case where a
> > leader
> > > is elected without a cluster id in meta.properties , since KRaft does
> not
> > > need cluster.id  in order to elect a leader.
> > >           In this case, the active controller will write a cluster id
> > > record during the bootstrap metadata write."
> > > Hmm, earlier, the KIP says "Upon discovering the cluster ID for the
> first
> > > time, these nodes need to persist this to meta.properties". Why do we
> > need
> > > to introduce a separate place to write the cluster id to
> > meta.properties.
> > >
> > > Jun
> > >
> > >
> > > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <ke...@gmail.com>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Manually bumping this thread after finalizing a design.
> > > > KIP link:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > >
> > > > Best,
> > > > Kevin Wu
> > > >
> > > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com>
> > wrote:
> > > >
> > > > > Hello all,
> > > > >
> > > > > I would like to start a discussion on KIP-1262, which proposes
> > removing
> > > > > the formatting requirement for brokers and observer controllers.
> > > > Currently,
> > > > > I am considering two high-level designs, and would appreciate
> > community
> > > > > feedback on both approaches to decide on a final design.
> > > > >
> > > > > KIP link:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > >
> > > > > Best,
> > > > > Kevin Wu
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Alyssa Huang via dev <de...@kafka.apache.org>]

Hey Kevin, thanks for the KIP!

AH1:

> The reason for this KIP is to remove the requirement of brokers needed to
> run the storage tool before starting Kafka.

I'd like to clarify whether you're saying the goal of the KIP is to remove
the need of running just the storage format command or the storage tool at
all. Also, I misunderstood

> Persisting this data does not need to be done before starting kafka.

 as saying we don't need to persist this data at all and that actually
messed up my understanding of the rest of your KIP on my first read - can
we change the wording to "Persisting this data does not have to be done
during storage formatting and can be done later during startup".

AH2: In the MV upgrade case, do we see any value in updating the
meta.properties file to v2?

AH3:

> Since this feature is associated with a new metadata record and
> MetadataVersion, broker bootstrapping with cluster ID is required on all
> MVs < X where X is the first MV that supports this feature. Because some
> MetadataVersion is resolved during each node's formatting, we can determine
> at format time if a ClusterIdRecord is needed as part of a controller's
> 0-0/bootstrap.checkpoint.

Can you go into some detail on how the MV is resolved during a node's
formatting? This might be interesting for readers to know in the
provisioning and re-formatting/adding a single node case.

AH4: Under Proposed Changes the first header is:

> meta.properties can be written during kafka broker/controller startup if
> it doesn't exist already (from formatting)

Nit, but should this be 'meta.properties' *will* be written during kafka
broker/controller startup if it doesn't exist already?

AH5: Could the following be expanded a bit? "write-path" might be too
generic since all nodes will write the MV they are formatted with to the
bootstrap metadata (but only on leader election will the elected node
attempt to write the bootstrap metadata MV to the metadata log). I found
"formatting a node (specifically controllers who can become leader)"
especially confusing, since I'm assuming you're saying the
write-to-metadata-log path for MV.

> This is enforceable along the write-path for MV, which occurs at two
> points: formatting a node (specifically controllers who can become leader)
> and upgrading the MV using kafka-features upgrade


AH6:

> However, kafka should still be able to handle the case where a leader is
> elected who does not have clusterId in meta.properties, which can occur if
> a majority of voters do not have a clusterId

The wording makes it sound like this is the exceptional case, but before I
read JR6 and your respond I had assumed this would be the normal case for a
cluster provisioned with KIP-1262 support. Can we make this more clear in
the KIP? Why are we making cluster id optional in meta.properties v2 if we
expect to still write it?

We can still enforce that bootstrap controllers must have formatted (and
> therefore persisted a cluster id to meta.properties) prior to starting
> kafka.

I'm confused if this is saying we will continue to enforce persisting
cluster id to meta.properties (given that you've also said kafka will
handle the case where a leader is elected w/o clusterid in its
meta.properties)

AH7:

> Nodes discover persist cluster id to meta.properties from metadata
> publishing pipeline
>
Seems like there's an extra word in this header

Thanks in advance for your responses :)
Alyssa

On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <ke...@gmail.com> wrote:

> Hi Jun,
>
> Thanks for the replies and questions.
>
> RE JR1: Updated the KIP with the record schema for ClusterIdRecord. One
> thing I'm not sure about yet is whether or not the record field should be
> of UUID or String type. This is because kafka's quickstart docs refer to
> setting `--cluster-id` to a UUID in the storage tool. However, many places
> in kafka broker/controller code (e.g. the raft client, broker lifecycle
> manager, and even the formatter itself) only require this type to be a
> String. Since not all Strings are valid UUIDs, making this record field of
> type UUID might be too restrictive and complicate upgrading the MV for
> existing clusters, since they might have a non-UUID cluster id string, but
> need to write this record when upgrading to an MV that supports this
> feature. Let me know what you think.
>
> RE JR2: Any controller node formatted with `--standalone,
> --initial-controllers` or who is part of the static voter set defined by
> `controller.quorum.voters` can write the ClusterIdRecord by including the
> `--cluster-id` argument to `kafka-storage format`. However, if the MV of
> the cluster supports it, there is exactly one writer of this record to the
> cluster metadata partition. The writer is the first active controller, who
> writes this record alongside other bootstrap metadata records (e.g.
> metadata version) during controller activation. At this point, we already
> depend on MV existing, since the active controller writes these bootstrap
> metadata records as a transaction if the MV supports it. I think writing
> the cluster id record would follow a similar pattern.
>
> RE JR3: When a node formats, it will write the meta.properties file. During
> formatting, a node must resolve the MV it wants to format with, which is
> explained more in RE JR5. I need to think about this more, but I think we
> should keep `--cluster-id` as a required flag for invoking the format
> command. If a broker/observer controller does not format, meta.properties
> is written without cluster id immediately after startup (i.e. where we read
> it from disk now in KafkaRaftServer).
>
> RE JR4: Yeah, will do. In this context, when I say observers I'm referring
> to any controllers who are not part of the KRaft voter set when they start
> kafka, or any brokers. I will make this explicit in the KIP. From the
> perspective of this feature and KRaft leader election, controller nodes who
> format with `--no-initial-controllers`, controller nodes who are not part
> of `controller.quorum.voters`, and brokers, all do not "need" to format,
> since they cannot become the active controller. This means they can resolve
> metadata like the cluster id after discovering the leader. We have a
> similar pattern with how controller nodes who format with
> `--no-initial-controllers` discover the kraft version of the cluster.
>
> RE JR5: If a node formats, it must resolve a metadata version with which to
> format. This comes from the `--release-version/--feature` flag and defaults
> to the latest production MV. Therefore, when a node formats with a metadata
> version that supports this feature, it will write the ClusterIdRecord to
> its `0-0/bootstrap.checkpoint`. If the node formats with a metadata version
> that does not support this feature, it does not write ClusterIdRecord to
> its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is assumed
> that this node is part of a cluster whose MV supports this. Otherwise, this
> is a misconfiguration and the node will fail to register with the leader
> since there is no way for it to persist cluster id to its meta.properties
> without formatting.
>
> Although I did not specify this yet on the KIP explicitly, after some
> offline discussion I think it makes sense to enforce the following
> invariant as part of the feature design: if the persisted metadata version
> supports this feature, the ClusterId record must also be persisted. This is
> enforceable on the write-path for MV, which occurs at two points-- during
> formatting and during feature upgrades. There is a similar pattern with
> kraft.version, as it gets written to disk at the same two points.
>
> RE JR6: The main motivation for writing cluster id to meta.properties as
> well is because it can act as a projection of the cluster metadata
> partition which essentially only exposes the cluster id to readers. For
> example, the raft layer needs to be aware of the cluster id for its own RPC
> handling/validation, but raft cannot read metadata records. There are many
> readers of this cluster id value during the startup of the cluster.
> Therefore, avoiding a read of the metadata partition to discover the value
> of this metadata will prevent more complications of the startup code.
>
> Best,
> Kevin Wu
>
>
> On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi, Kevin,
> >
> > Thanks for the KIP. A few comments.
> >
> > JR1. ClusterIdRecord : Could you define the record format?
> >
> > JR2. "a new MetadataVersion that supports encoding/decoding this record.
> > This means that during formatting, the bootstrap ClusterIdRecord is only
> > written if the cluster is formatted with a MV that supports this
> feature."
> > Could you describe who writes the ClusterIdRecord? Is it the leader
> > controller? Also, when is the record written? Do we guarantee that MV is
> > available at that time?
> >
> > JR3. "meta.properties can be written during kafka broker/controller
> startup
> > if it doesn't exist already (from formatting)"
> > Could you describe when meta.properties is written? Is MV available at
> that
> > time?
> >
> > JR4. "Introduce a metadata record for cluster id + observers persist
> > cluster id to meta.properties from metadata publishing pipeline"
> > Could you clarify what observers are? Are they observer controllers or
> are
> > they brokers (which are referred to as observers to the controller)?
> >
> > JR5. "Bootstrap controllers can add a mandatory “cluster id” record
> during
> > formatting"
> > This sounds like adding a ClusterIdRecord is optional. If so, could you
> > describe when a record will be added and when a record will not be added?
> >
> > JR6. "However, kafka should still be able to handle the case where a
> leader
> > is elected without a cluster id in meta.properties , since KRaft does not
> > need cluster.id  in order to elect a leader.
> >           In this case, the active controller will write a cluster id
> > record during the bootstrap metadata write."
> > Hmm, earlier, the KIP says "Upon discovering the cluster ID for the first
> > time, these nodes need to persist this to meta.properties". Why do we
> need
> > to introduce a separate place to write the cluster id to
> meta.properties.
> >
> > Jun
> >
> >
> > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <ke...@gmail.com>
> wrote:
> >
> > > Hi all,
> > >
> > > Manually bumping this thread after finalizing a design.
> > > KIP link:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > >
> > > Best,
> > > Kevin Wu
> > >
> > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com>
> wrote:
> > >
> > > > Hello all,
> > > >
> > > > I would like to start a discussion on KIP-1262, which proposes
> removing
> > > > the formatting requirement for brokers and observer controllers.
> > > Currently,
> > > > I am considering two high-level designs, and would appreciate
> community
> > > > feedback on both approaches to decide on a final design.
> > > >
> > > > KIP link:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > >
> > > > Best,
> > > > Kevin Wu
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Luke Chen <sh...@gmail.com>]

Hi Kevin,

Thanks for the response.

LC2: OK, that matches what I thought. I think it's fine because in ZK's
era, the brokers also rely on the `zookeeper.connect` config to get the all
cluster metadata. It seems it never caused brokers to connect to the wrong
ZK cluster.

LC4: I see. Thanks.

Luke

On Fri, Mar 13, 2026 at 2:54 PM Kevin Wu <ke...@gmail.com> wrote:

> Hi Luke,
>
> Thanks for the reply.
>
> RE LC1 + LC3: Sure, I will update the KIP.
>
> RE LC2: Yes, when a node does not have a cluster.id set, it must discover
> it via Fetch. This means that a node can talk to the "wrong" cluster at
> this point. However, upon discovering and persisting a cluster.id, a node
> will not be able to successfully fetch from a KRaft leader with a different
> cluster.id. The only way to prevent this case would be to format the
> node with a cluster id.
>
> RE LC4: It is not possible for meta.properties V1 to exist without a
> cluster.id. Look at MetaProperties.Builder#build() and
> MetaPropertiesEnsemble#verify().
>
> Best,
> Kevin Wu
>
> On Thu, Mar 12, 2026 at 9:48 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Kevin,
> >
> > Thanks for the KIP.
> >
> > Comments:
> > LC1. It's good if we can display the schema of meta.properties v2, and
> > what's the difference from v1, like we showed for API change.
> >
> > LC2. Before this KIP, the formatting nodes with the correct cluster ID
> > makes sure the brokers/observer controllers will talk to the expected
> > controllers. But after this KIP, it's possible that the brokers/observer
> > controllers without formatting could connect to the wrong cluster
> > controllers, and get discover the cluster ID from fetch response and
> > persist it. Is this correct? Looks like this is the risk users need to
> take
> > if they don't want to format them?
> >
> > LC3. "If this ID doesn't match the KRaft leader's the leader will reject
> > requests from the node"
> > There is a missing word in the sentence.
> >
> > LC4. "If meta.properties exists without a cluster.id and is V2, it will
> be
> > discovered later (described below)"
> > Is it possible the meta.properties exists without a cluster.id but it's
> in
> > V1? If so, how will we handle it?
> >
> > Thank you,
> > Luke
> >
> > On Wed, Mar 4, 2026 at 2:39 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> > > Hi, Kevin,
> > >
> > > Thanks for the reply. The KIP looks good to me now.
> > >
> > > Jun
> > >
> > > On Tue, Mar 3, 2026 at 4:54 AM Kevin Wu <ke...@gmail.com>
> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > RE JR1: "If an existing string can't be converted to uuid, we can
> fail
> > > the
> > > > node. This shouldn't happen for a well formatted cluster, right?"
> > > > Currently, you can format a cluster with a non-UUID cluster ID
> string,
> > > and
> > > > kafka considers this "well-formatted" (i.e. formatting code accepts
> > > String,
> > > > server startup works, and clusterId is a String in-memory etc.). Our
> > > > documentation references formatting with a UUID cluster id generated
> > via
> > > > `kafka-storage random-uuid`, but this is not a requirement in the
> code.
> > > If
> > > > we make this record have a UUID to be consistent with TopicRecord, it
> > is
> > > > not clear to me what the MV upgrade path is for existing clusters who
> > > > formatted `meta.properties` with a non-UUID String. We have to write
> a
> > > new
> > > > UUID cluster id, which violates the invariant that the cluster id
> > cannot
> > > > change over the lifetime of a cluster.
> > > >
> > > > RE JR6: I plan on still requiring bootstrap controllers to format.
> This
> > > > means we should not expect a leader to be elected who does not have a
> > > > cluster id. Bootstrap controllers will fail when reading in
> > > meta.properties
> > > > in KafkaRaftServer. I will remove this section.
> > > >
> > > > RE JR7: Apologies, I mixed up the numbers with another KIP.
> > > >
> > > > RE JR8:
> > > > For brokers, the readers of cluster id during startup are the
> > > > BrokerLifecycleManager, KafkaApis, DynamicTopicClusterQuotaPublisher,
> > and
> > > > endpointReadyFutures. It is okay to block startup on fetching the
> > cluster
> > > > id from KRaft, since we also block startup on broker lifecycle
> manager
> > > > initial catch up future. Discovering the cluster id value for the
> first
> > > > time would only require a single FetchSnapshot or a Fetch of the
> > > bootstrap
> > > > metadata records.
> > > >
> > > > For controllers, the readers are endpointReadyFutures,
> > > > QuorumController, ControllerApis, ControllerRegistrationManager, and
> > > > DynamicTopicClusterQuotaPublisher. For bootstrap controllers, this
> > > blocking
> > > > does not occur. For observers, they are essentially brokers from the
> > > > perspective of KRaft, so I think it is okay to block even the
> > > > initialization of QuorumController until the cluster id is
> discovered.
> > > Just
> > > > like with brokers, we only block for 1 successful Fetch/Fetch
> Snapshot
> > > loop
> > > > until this data is known. One detail is that for auto-joining
> observers
> > > in
> > > > kraft.version=1, they need to wait until they persist cluster id
> before
> > > > they try to join the voter set.
> > > >
> > > > RE JR9.1:
> > > > This can also mean the broker skipped formatting, and does not have a
> > > > cluster id. In this case, it will persist cluster id to
> > meta.properties.
> > > >
> > > > The other case is when the broker has a cluster.id in
> meta.properties.
> > > In
> > > > this case, the broker cannot discover a different cluster id via a
> > > > ClusterIdRecord in FetchResponse. In fact, the broker will not be
> able
> > to
> > > > successfully complete any KRaft RPCs against the leader. For the
> broker
> > > to
> > > > receive a non-error FetchResponse with metadata records (which would
> be
> > > the
> > > > only way to learn of a different ClusterIdRecord), the KRaft leader
> > > checks
> > > > that the request cluster id is absent, or that the request cluster id
> > > > matches its own (which is the cluster id in its
> > > > meta.properties/ClusterIdRecord if the invariant I mentioned in my
> > > previous
> > > > message is enforced properly). This case could happen when bootstrap
> > > > endpoints point to the wrong cluster during restart of a node. The
> > logic
> > > > above would result in startup timing out and shutting down the node
> > > because
> > > > the local node is not able to participate in KRaft for another
> cluster.
> > > >
> > > > RE JR9.2: Yes, the broker's startup will eventually timeout and fail.
> > The
> > > > broker won't have cluster.id in meta.properties, and the cluster
> > cannot
> > > > send the broker a cluster id via ClusterIdRecord. The same would
> apply
> > > for
> > > > an observer controller. This is a misconfiguration in my opinion.
> > > >
> > > > On Tue, Mar 3, 2026 at 12:22 AM Jun Rao via dev <
> dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hi, Kevin,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > JR1. ClusterIdRecord:
> > > > > It would be better for ClusterId to have the type uuid. This will
> > make
> > > it
> > > > > consistent with topicId in TopicRecord. If an existing string can't
> > be
> > > > > converted to uuid, we can fail the node. This shouldn't happen for
> a
> > > well
> > > > > formatted cluster, right?
> > > > >
> > > > > JR6. Have you decided what to include in this KIP? If this KIP
> still
> > > > > requires the formatting for bootstrap controllers, what's described
> > > here
> > > > > can't happen.
> > > > >
> > > > > JR7. "After KIP-1286, kafka operators no longer need to format all
> > > nodes"
> > > > > KIP-1286 seems to be the wrong KIP?
> > > > >
> > > > > JR8. "The readers of cluster id initialized during startup can wait
> > for
> > > > > both the above before being initialized."
> > > > > What are those readers? Are they ok to block?
> > > > >
> > > > > JR9. A couple more upgrade scenarios.
> > > > > JR9.1 If the MV has been bumped, after a broker starts up, it
> > discovers
> > > > > that the clusterId in ClusterIdRecord doesn't match the one in
> > > > > meta.properties. Will the broker fail?
> > > > > JR9.2 If the MV hasn't been bumped, a new broker with the new
> version
> > > of
> > > > > the software is started without formatting, will it fail during
> > > startup?
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <ke...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Thanks for the replies and questions.
> > > > > >
> > > > > > RE JR1: Updated the KIP with the record schema for
> ClusterIdRecord.
> > > One
> > > > > > thing I'm not sure about yet is whether or not the record field
> > > should
> > > > be
> > > > > > of UUID or String type. This is because kafka's quickstart docs
> > refer
> > > > to
> > > > > > setting `--cluster-id` to a UUID in the storage tool. However,
> many
> > > > > places
> > > > > > in kafka broker/controller code (e.g. the raft client, broker
> > > lifecycle
> > > > > > manager, and even the formatter itself) only require this type to
> > be
> > > a
> > > > > > String. Since not all Strings are valid UUIDs, making this record
> > > field
> > > > > of
> > > > > > type UUID might be too restrictive and complicate upgrading the
> MV
> > > for
> > > > > > existing clusters, since they might have a non-UUID cluster id
> > > string,
> > > > > but
> > > > > > need to write this record when upgrading to an MV that supports
> > this
> > > > > > feature. Let me know what you think.
> > > > > >
> > > > > > RE JR2: Any controller node formatted with `--standalone,
> > > > > > --initial-controllers` or who is part of the static voter set
> > defined
> > > > by
> > > > > > `controller.quorum.voters` can write the ClusterIdRecord by
> > including
> > > > the
> > > > > > `--cluster-id` argument to `kafka-storage format`. However, if
> the
> > MV
> > > > of
> > > > > > the cluster supports it, there is exactly one writer of this
> record
> > > to
> > > > > the
> > > > > > cluster metadata partition. The writer is the first active
> > > controller,
> > > > > who
> > > > > > writes this record alongside other bootstrap metadata records
> (e.g.
> > > > > > metadata version) during controller activation. At this point, we
> > > > already
> > > > > > depend on MV existing, since the active controller writes these
> > > > bootstrap
> > > > > > metadata records as a transaction if the MV supports it. I think
> > > > writing
> > > > > > the cluster id record would follow a similar pattern.
> > > > > >
> > > > > > RE JR3: When a node formats, it will write the meta.properties
> > file.
> > > > > During
> > > > > > formatting, a node must resolve the MV it wants to format with,
> > which
> > > > is
> > > > > > explained more in RE JR5. I need to think about this more, but I
> > > think
> > > > we
> > > > > > should keep `--cluster-id` as a required flag for invoking the
> > format
> > > > > > command. If a broker/observer controller does not format,
> > > > meta.properties
> > > > > > is written without cluster id immediately after startup (i.e.
> where
> > > we
> > > > > read
> > > > > > it from disk now in KafkaRaftServer).
> > > > > >
> > > > > > RE JR4: Yeah, will do. In this context, when I say observers I'm
> > > > > referring
> > > > > > to any controllers who are not part of the KRaft voter set when
> > they
> > > > > start
> > > > > > kafka, or any brokers. I will make this explicit in the KIP. From
> > the
> > > > > > perspective of this feature and KRaft leader election, controller
> > > nodes
> > > > > who
> > > > > > format with `--no-initial-controllers`, controller nodes who are
> > not
> > > > part
> > > > > > of `controller.quorum.voters`, and brokers, all do not "need" to
> > > > format,
> > > > > > since they cannot become the active controller. This means they
> can
> > > > > resolve
> > > > > > metadata like the cluster id after discovering the leader. We
> have
> > a
> > > > > > similar pattern with how controller nodes who format with
> > > > > > `--no-initial-controllers` discover the kraft version of the
> > cluster.
> > > > > >
> > > > > > RE JR5: If a node formats, it must resolve a metadata version
> with
> > > > which
> > > > > to
> > > > > > format. This comes from the `--release-version/--feature` flag
> and
> > > > > defaults
> > > > > > to the latest production MV. Therefore, when a node formats with
> a
> > > > > metadata
> > > > > > version that supports this feature, it will write the
> > ClusterIdRecord
> > > > to
> > > > > > its `0-0/bootstrap.checkpoint`. If the node formats with a
> metadata
> > > > > version
> > > > > > that does not support this feature, it does not write
> > ClusterIdRecord
> > > > to
> > > > > > its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is
> > > > assumed
> > > > > > that this node is part of a cluster whose MV supports this.
> > > Otherwise,
> > > > > this
> > > > > > is a misconfiguration and the node will fail to register with the
> > > > leader
> > > > > > since there is no way for it to persist cluster id to its
> > > > meta.properties
> > > > > > without formatting.
> > > > > >
> > > > > > Although I did not specify this yet on the KIP explicitly, after
> > some
> > > > > > offline discussion I think it makes sense to enforce the
> following
> > > > > > invariant as part of the feature design: if the persisted
> metadata
> > > > > version
> > > > > > supports this feature, the ClusterId record must also be
> persisted.
> > > > This
> > > > > is
> > > > > > enforceable on the write-path for MV, which occurs at two
> points--
> > > > during
> > > > > > formatting and during feature upgrades. There is a similar
> pattern
> > > with
> > > > > > kraft.version, as it gets written to disk at the same two points.
> > > > > >
> > > > > > RE JR6: The main motivation for writing cluster id to
> > meta.properties
> > > > as
> > > > > > well is because it can act as a projection of the cluster
> metadata
> > > > > > partition which essentially only exposes the cluster id to
> readers.
> > > For
> > > > > > example, the raft layer needs to be aware of the cluster id for
> its
> > > own
> > > > > RPC
> > > > > > handling/validation, but raft cannot read metadata records. There
> > are
> > > > > many
> > > > > > readers of this cluster id value during the startup of the
> cluster.
> > > > > > Therefore, avoiding a read of the metadata partition to discover
> > the
> > > > > value
> > > > > > of this metadata will prevent more complications of the startup
> > code.
> > > > > >
> > > > > > Best,
> > > > > > Kevin Wu
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <
> > > dev@kafka.apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi, Kevin,
> > > > > > >
> > > > > > > Thanks for the KIP. A few comments.
> > > > > > >
> > > > > > > JR1. ClusterIdRecord : Could you define the record format?
> > > > > > >
> > > > > > > JR2. "a new MetadataVersion that supports encoding/decoding
> this
> > > > > record.
> > > > > > > This means that during formatting, the bootstrap
> ClusterIdRecord
> > is
> > > > > only
> > > > > > > written if the cluster is formatted with a MV that supports
> this
> > > > > > feature."
> > > > > > > Could you describe who writes the ClusterIdRecord? Is it the
> > leader
> > > > > > > controller? Also, when is the record written? Do we guarantee
> > that
> > > MV
> > > > > is
> > > > > > > available at that time?
> > > > > > >
> > > > > > > JR3. "meta.properties can be written during kafka
> > broker/controller
> > > > > > startup
> > > > > > > if it doesn't exist already (from formatting)"
> > > > > > > Could you describe when meta.properties is written? Is MV
> > available
> > > > at
> > > > > > that
> > > > > > > time?
> > > > > > >
> > > > > > > JR4. "Introduce a metadata record for cluster id + observers
> > > persist
> > > > > > > cluster id to meta.properties from metadata publishing
> pipeline"
> > > > > > > Could you clarify what observers are? Are they observer
> > controllers
> > > > or
> > > > > > are
> > > > > > > they brokers (which are referred to as observers to the
> > > controller)?
> > > > > > >
> > > > > > > JR5. "Bootstrap controllers can add a mandatory “cluster id”
> > record
> > > > > > during
> > > > > > > formatting"
> > > > > > > This sounds like adding a ClusterIdRecord is optional. If so,
> > could
> > > > you
> > > > > > > describe when a record will be added and when a record will not
> > be
> > > > > added?
> > > > > > >
> > > > > > > JR6. "However, kafka should still be able to handle the case
> > where
> > > a
> > > > > > leader
> > > > > > > is elected without a cluster id in meta.properties , since
> KRaft
> > > does
> > > > > not
> > > > > > > need cluster.id  in order to elect a leader.
> > > > > > >           In this case, the active controller will write a
> > cluster
> > > id
> > > > > > > record during the bootstrap metadata write."
> > > > > > > Hmm, earlier, the KIP says "Upon discovering the cluster ID for
> > the
> > > > > first
> > > > > > > time, these nodes need to persist this to meta.properties". Why
> > do
> > > we
> > > > > > need
> > > > > > > to introduce a separate place to write the cluster id to
> > > > > > meta.properties.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <
> > kevin.wu2412@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Manually bumping this thread after finalizing a design.
> > > > > > > > KIP link:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Kevin Wu
> > > > > > > >
> > > > > > > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <
> > kevin.wu2412@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello all,
> > > > > > > > >
> > > > > > > > > I would like to start a discussion on KIP-1262, which
> > proposes
> > > > > > removing
> > > > > > > > > the formatting requirement for brokers and observer
> > > controllers.
> > > > > > > > Currently,
> > > > > > > > > I am considering two high-level designs, and would
> appreciate
> > > > > > community
> > > > > > > > > feedback on both approaches to decide on a final design.
> > > > > > > > >
> > > > > > > > > KIP link:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Kevin Wu
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi Luke,

Thanks for the reply.

RE LC1 + LC3: Sure, I will update the KIP.

RE LC2: Yes, when a node does not have a cluster.id set, it must discover
it via Fetch. This means that a node can talk to the "wrong" cluster at
this point. However, upon discovering and persisting a cluster.id, a node
will not be able to successfully fetch from a KRaft leader with a different
cluster.id. The only way to prevent this case would be to format the
node with a cluster id.

RE LC4: It is not possible for meta.properties V1 to exist without a
cluster.id. Look at MetaProperties.Builder#build() and
MetaPropertiesEnsemble#verify().

Best,
Kevin Wu

On Thu, Mar 12, 2026 at 9:48 AM Luke Chen <sh...@gmail.com> wrote:

> Hi Kevin,
>
> Thanks for the KIP.
>
> Comments:
> LC1. It's good if we can display the schema of meta.properties v2, and
> what's the difference from v1, like we showed for API change.
>
> LC2. Before this KIP, the formatting nodes with the correct cluster ID
> makes sure the brokers/observer controllers will talk to the expected
> controllers. But after this KIP, it's possible that the brokers/observer
> controllers without formatting could connect to the wrong cluster
> controllers, and get discover the cluster ID from fetch response and
> persist it. Is this correct? Looks like this is the risk users need to take
> if they don't want to format them?
>
> LC3. "If this ID doesn't match the KRaft leader's the leader will reject
> requests from the node"
> There is a missing word in the sentence.
>
> LC4. "If meta.properties exists without a cluster.id and is V2, it will be
> discovered later (described below)"
> Is it possible the meta.properties exists without a cluster.id but it's in
> V1? If so, how will we handle it?
>
> Thank you,
> Luke
>
> On Wed, Mar 4, 2026 at 2:39 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi, Kevin,
> >
> > Thanks for the reply. The KIP looks good to me now.
> >
> > Jun
> >
> > On Tue, Mar 3, 2026 at 4:54 AM Kevin Wu <ke...@gmail.com> wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for the reply.
> > >
> > > RE JR1: "If an existing string can't be converted to uuid, we can fail
> > the
> > > node. This shouldn't happen for a well formatted cluster, right?"
> > > Currently, you can format a cluster with a non-UUID cluster ID string,
> > and
> > > kafka considers this "well-formatted" (i.e. formatting code accepts
> > String,
> > > server startup works, and clusterId is a String in-memory etc.). Our
> > > documentation references formatting with a UUID cluster id generated
> via
> > > `kafka-storage random-uuid`, but this is not a requirement in the code.
> > If
> > > we make this record have a UUID to be consistent with TopicRecord, it
> is
> > > not clear to me what the MV upgrade path is for existing clusters who
> > > formatted `meta.properties` with a non-UUID String. We have to write a
> > new
> > > UUID cluster id, which violates the invariant that the cluster id
> cannot
> > > change over the lifetime of a cluster.
> > >
> > > RE JR6: I plan on still requiring bootstrap controllers to format. This
> > > means we should not expect a leader to be elected who does not have a
> > > cluster id. Bootstrap controllers will fail when reading in
> > meta.properties
> > > in KafkaRaftServer. I will remove this section.
> > >
> > > RE JR7: Apologies, I mixed up the numbers with another KIP.
> > >
> > > RE JR8:
> > > For brokers, the readers of cluster id during startup are the
> > > BrokerLifecycleManager, KafkaApis, DynamicTopicClusterQuotaPublisher,
> and
> > > endpointReadyFutures. It is okay to block startup on fetching the
> cluster
> > > id from KRaft, since we also block startup on broker lifecycle manager
> > > initial catch up future. Discovering the cluster id value for the first
> > > time would only require a single FetchSnapshot or a Fetch of the
> > bootstrap
> > > metadata records.
> > >
> > > For controllers, the readers are endpointReadyFutures,
> > > QuorumController, ControllerApis, ControllerRegistrationManager, and
> > > DynamicTopicClusterQuotaPublisher. For bootstrap controllers, this
> > blocking
> > > does not occur. For observers, they are essentially brokers from the
> > > perspective of KRaft, so I think it is okay to block even the
> > > initialization of QuorumController until the cluster id is discovered.
> > Just
> > > like with brokers, we only block for 1 successful Fetch/Fetch Snapshot
> > loop
> > > until this data is known. One detail is that for auto-joining observers
> > in
> > > kraft.version=1, they need to wait until they persist cluster id before
> > > they try to join the voter set.
> > >
> > > RE JR9.1:
> > > This can also mean the broker skipped formatting, and does not have a
> > > cluster id. In this case, it will persist cluster id to
> meta.properties.
> > >
> > > The other case is when the broker has a cluster.id in meta.properties.
> > In
> > > this case, the broker cannot discover a different cluster id via a
> > > ClusterIdRecord in FetchResponse. In fact, the broker will not be able
> to
> > > successfully complete any KRaft RPCs against the leader. For the broker
> > to
> > > receive a non-error FetchResponse with metadata records (which would be
> > the
> > > only way to learn of a different ClusterIdRecord), the KRaft leader
> > checks
> > > that the request cluster id is absent, or that the request cluster id
> > > matches its own (which is the cluster id in its
> > > meta.properties/ClusterIdRecord if the invariant I mentioned in my
> > previous
> > > message is enforced properly). This case could happen when bootstrap
> > > endpoints point to the wrong cluster during restart of a node. The
> logic
> > > above would result in startup timing out and shutting down the node
> > because
> > > the local node is not able to participate in KRaft for another cluster.
> > >
> > > RE JR9.2: Yes, the broker's startup will eventually timeout and fail.
> The
> > > broker won't have cluster.id in meta.properties, and the cluster
> cannot
> > > send the broker a cluster id via ClusterIdRecord. The same would apply
> > for
> > > an observer controller. This is a misconfiguration in my opinion.
> > >
> > > On Tue, Mar 3, 2026 at 12:22 AM Jun Rao via dev <de...@kafka.apache.org>
> > > wrote:
> > >
> > > > Hi, Kevin,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > JR1. ClusterIdRecord:
> > > > It would be better for ClusterId to have the type uuid. This will
> make
> > it
> > > > consistent with topicId in TopicRecord. If an existing string can't
> be
> > > > converted to uuid, we can fail the node. This shouldn't happen for a
> > well
> > > > formatted cluster, right?
> > > >
> > > > JR6. Have you decided what to include in this KIP? If this KIP still
> > > > requires the formatting for bootstrap controllers, what's described
> > here
> > > > can't happen.
> > > >
> > > > JR7. "After KIP-1286, kafka operators no longer need to format all
> > nodes"
> > > > KIP-1286 seems to be the wrong KIP?
> > > >
> > > > JR8. "The readers of cluster id initialized during startup can wait
> for
> > > > both the above before being initialized."
> > > > What are those readers? Are they ok to block?
> > > >
> > > > JR9. A couple more upgrade scenarios.
> > > > JR9.1 If the MV has been bumped, after a broker starts up, it
> discovers
> > > > that the clusterId in ClusterIdRecord doesn't match the one in
> > > > meta.properties. Will the broker fail?
> > > > JR9.2 If the MV hasn't been bumped, a new broker with the new version
> > of
> > > > the software is started without formatting, will it fail during
> > startup?
> > > >
> > > > Jun
> > > >
> > > > On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <ke...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks for the replies and questions.
> > > > >
> > > > > RE JR1: Updated the KIP with the record schema for ClusterIdRecord.
> > One
> > > > > thing I'm not sure about yet is whether or not the record field
> > should
> > > be
> > > > > of UUID or String type. This is because kafka's quickstart docs
> refer
> > > to
> > > > > setting `--cluster-id` to a UUID in the storage tool. However, many
> > > > places
> > > > > in kafka broker/controller code (e.g. the raft client, broker
> > lifecycle
> > > > > manager, and even the formatter itself) only require this type to
> be
> > a
> > > > > String. Since not all Strings are valid UUIDs, making this record
> > field
> > > > of
> > > > > type UUID might be too restrictive and complicate upgrading the MV
> > for
> > > > > existing clusters, since they might have a non-UUID cluster id
> > string,
> > > > but
> > > > > need to write this record when upgrading to an MV that supports
> this
> > > > > feature. Let me know what you think.
> > > > >
> > > > > RE JR2: Any controller node formatted with `--standalone,
> > > > > --initial-controllers` or who is part of the static voter set
> defined
> > > by
> > > > > `controller.quorum.voters` can write the ClusterIdRecord by
> including
> > > the
> > > > > `--cluster-id` argument to `kafka-storage format`. However, if the
> MV
> > > of
> > > > > the cluster supports it, there is exactly one writer of this record
> > to
> > > > the
> > > > > cluster metadata partition. The writer is the first active
> > controller,
> > > > who
> > > > > writes this record alongside other bootstrap metadata records (e.g.
> > > > > metadata version) during controller activation. At this point, we
> > > already
> > > > > depend on MV existing, since the active controller writes these
> > > bootstrap
> > > > > metadata records as a transaction if the MV supports it. I think
> > > writing
> > > > > the cluster id record would follow a similar pattern.
> > > > >
> > > > > RE JR3: When a node formats, it will write the meta.properties
> file.
> > > > During
> > > > > formatting, a node must resolve the MV it wants to format with,
> which
> > > is
> > > > > explained more in RE JR5. I need to think about this more, but I
> > think
> > > we
> > > > > should keep `--cluster-id` as a required flag for invoking the
> format
> > > > > command. If a broker/observer controller does not format,
> > > meta.properties
> > > > > is written without cluster id immediately after startup (i.e. where
> > we
> > > > read
> > > > > it from disk now in KafkaRaftServer).
> > > > >
> > > > > RE JR4: Yeah, will do. In this context, when I say observers I'm
> > > > referring
> > > > > to any controllers who are not part of the KRaft voter set when
> they
> > > > start
> > > > > kafka, or any brokers. I will make this explicit in the KIP. From
> the
> > > > > perspective of this feature and KRaft leader election, controller
> > nodes
> > > > who
> > > > > format with `--no-initial-controllers`, controller nodes who are
> not
> > > part
> > > > > of `controller.quorum.voters`, and brokers, all do not "need" to
> > > format,
> > > > > since they cannot become the active controller. This means they can
> > > > resolve
> > > > > metadata like the cluster id after discovering the leader. We have
> a
> > > > > similar pattern with how controller nodes who format with
> > > > > `--no-initial-controllers` discover the kraft version of the
> cluster.
> > > > >
> > > > > RE JR5: If a node formats, it must resolve a metadata version with
> > > which
> > > > to
> > > > > format. This comes from the `--release-version/--feature` flag and
> > > > defaults
> > > > > to the latest production MV. Therefore, when a node formats with a
> > > > metadata
> > > > > version that supports this feature, it will write the
> ClusterIdRecord
> > > to
> > > > > its `0-0/bootstrap.checkpoint`. If the node formats with a metadata
> > > > version
> > > > > that does not support this feature, it does not write
> ClusterIdRecord
> > > to
> > > > > its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is
> > > assumed
> > > > > that this node is part of a cluster whose MV supports this.
> > Otherwise,
> > > > this
> > > > > is a misconfiguration and the node will fail to register with the
> > > leader
> > > > > since there is no way for it to persist cluster id to its
> > > meta.properties
> > > > > without formatting.
> > > > >
> > > > > Although I did not specify this yet on the KIP explicitly, after
> some
> > > > > offline discussion I think it makes sense to enforce the following
> > > > > invariant as part of the feature design: if the persisted metadata
> > > > version
> > > > > supports this feature, the ClusterId record must also be persisted.
> > > This
> > > > is
> > > > > enforceable on the write-path for MV, which occurs at two points--
> > > during
> > > > > formatting and during feature upgrades. There is a similar pattern
> > with
> > > > > kraft.version, as it gets written to disk at the same two points.
> > > > >
> > > > > RE JR6: The main motivation for writing cluster id to
> meta.properties
> > > as
> > > > > well is because it can act as a projection of the cluster metadata
> > > > > partition which essentially only exposes the cluster id to readers.
> > For
> > > > > example, the raft layer needs to be aware of the cluster id for its
> > own
> > > > RPC
> > > > > handling/validation, but raft cannot read metadata records. There
> are
> > > > many
> > > > > readers of this cluster id value during the startup of the cluster.
> > > > > Therefore, avoiding a read of the metadata partition to discover
> the
> > > > value
> > > > > of this metadata will prevent more complications of the startup
> code.
> > > > >
> > > > > Best,
> > > > > Kevin Wu
> > > > >
> > > > >
> > > > > On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <
> > dev@kafka.apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi, Kevin,
> > > > > >
> > > > > > Thanks for the KIP. A few comments.
> > > > > >
> > > > > > JR1. ClusterIdRecord : Could you define the record format?
> > > > > >
> > > > > > JR2. "a new MetadataVersion that supports encoding/decoding this
> > > > record.
> > > > > > This means that during formatting, the bootstrap ClusterIdRecord
> is
> > > > only
> > > > > > written if the cluster is formatted with a MV that supports this
> > > > > feature."
> > > > > > Could you describe who writes the ClusterIdRecord? Is it the
> leader
> > > > > > controller? Also, when is the record written? Do we guarantee
> that
> > MV
> > > > is
> > > > > > available at that time?
> > > > > >
> > > > > > JR3. "meta.properties can be written during kafka
> broker/controller
> > > > > startup
> > > > > > if it doesn't exist already (from formatting)"
> > > > > > Could you describe when meta.properties is written? Is MV
> available
> > > at
> > > > > that
> > > > > > time?
> > > > > >
> > > > > > JR4. "Introduce a metadata record for cluster id + observers
> > persist
> > > > > > cluster id to meta.properties from metadata publishing pipeline"
> > > > > > Could you clarify what observers are? Are they observer
> controllers
> > > or
> > > > > are
> > > > > > they brokers (which are referred to as observers to the
> > controller)?
> > > > > >
> > > > > > JR5. "Bootstrap controllers can add a mandatory “cluster id”
> record
> > > > > during
> > > > > > formatting"
> > > > > > This sounds like adding a ClusterIdRecord is optional. If so,
> could
> > > you
> > > > > > describe when a record will be added and when a record will not
> be
> > > > added?
> > > > > >
> > > > > > JR6. "However, kafka should still be able to handle the case
> where
> > a
> > > > > leader
> > > > > > is elected without a cluster id in meta.properties , since KRaft
> > does
> > > > not
> > > > > > need cluster.id  in order to elect a leader.
> > > > > >           In this case, the active controller will write a
> cluster
> > id
> > > > > > record during the bootstrap metadata write."
> > > > > > Hmm, earlier, the KIP says "Upon discovering the cluster ID for
> the
> > > > first
> > > > > > time, these nodes need to persist this to meta.properties". Why
> do
> > we
> > > > > need
> > > > > > to introduce a separate place to write the cluster id to
> > > > > meta.properties.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <
> kevin.wu2412@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Manually bumping this thread after finalizing a design.
> > > > > > > KIP link:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > > > >
> > > > > > > Best,
> > > > > > > Kevin Wu
> > > > > > >
> > > > > > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <
> kevin.wu2412@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello all,
> > > > > > > >
> > > > > > > > I would like to start a discussion on KIP-1262, which
> proposes
> > > > > removing
> > > > > > > > the formatting requirement for brokers and observer
> > controllers.
> > > > > > > Currently,
> > > > > > > > I am considering two high-level designs, and would appreciate
> > > > > community
> > > > > > > > feedback on both approaches to decide on a final design.
> > > > > > > >
> > > > > > > > KIP link:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Kevin Wu
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Luke Chen <sh...@gmail.com>]

Hi Kevin,

Thanks for the KIP.

Comments:
LC1. It's good if we can display the schema of meta.properties v2, and
what's the difference from v1, like we showed for API change.

LC2. Before this KIP, the formatting nodes with the correct cluster ID
makes sure the brokers/observer controllers will talk to the expected
controllers. But after this KIP, it's possible that the brokers/observer
controllers without formatting could connect to the wrong cluster
controllers, and get discover the cluster ID from fetch response and
persist it. Is this correct? Looks like this is the risk users need to take
if they don't want to format them?

LC3. "If this ID doesn't match the KRaft leader's the leader will reject
requests from the node"
There is a missing word in the sentence.

LC4. "If meta.properties exists without a cluster.id and is V2, it will be
discovered later (described below)"
Is it possible the meta.properties exists without a cluster.id but it's in
V1? If so, how will we handle it?

Thank you,
Luke

On Wed, Mar 4, 2026 at 2:39 AM Jun Rao via dev <de...@kafka.apache.org> wrote:

> Hi, Kevin,
>
> Thanks for the reply. The KIP looks good to me now.
>
> Jun
>
> On Tue, Mar 3, 2026 at 4:54 AM Kevin Wu <ke...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the reply.
> >
> > RE JR1: "If an existing string can't be converted to uuid, we can fail
> the
> > node. This shouldn't happen for a well formatted cluster, right?"
> > Currently, you can format a cluster with a non-UUID cluster ID string,
> and
> > kafka considers this "well-formatted" (i.e. formatting code accepts
> String,
> > server startup works, and clusterId is a String in-memory etc.). Our
> > documentation references formatting with a UUID cluster id generated via
> > `kafka-storage random-uuid`, but this is not a requirement in the code.
> If
> > we make this record have a UUID to be consistent with TopicRecord, it is
> > not clear to me what the MV upgrade path is for existing clusters who
> > formatted `meta.properties` with a non-UUID String. We have to write a
> new
> > UUID cluster id, which violates the invariant that the cluster id cannot
> > change over the lifetime of a cluster.
> >
> > RE JR6: I plan on still requiring bootstrap controllers to format. This
> > means we should not expect a leader to be elected who does not have a
> > cluster id. Bootstrap controllers will fail when reading in
> meta.properties
> > in KafkaRaftServer. I will remove this section.
> >
> > RE JR7: Apologies, I mixed up the numbers with another KIP.
> >
> > RE JR8:
> > For brokers, the readers of cluster id during startup are the
> > BrokerLifecycleManager, KafkaApis, DynamicTopicClusterQuotaPublisher, and
> > endpointReadyFutures. It is okay to block startup on fetching the cluster
> > id from KRaft, since we also block startup on broker lifecycle manager
> > initial catch up future. Discovering the cluster id value for the first
> > time would only require a single FetchSnapshot or a Fetch of the
> bootstrap
> > metadata records.
> >
> > For controllers, the readers are endpointReadyFutures,
> > QuorumController, ControllerApis, ControllerRegistrationManager, and
> > DynamicTopicClusterQuotaPublisher. For bootstrap controllers, this
> blocking
> > does not occur. For observers, they are essentially brokers from the
> > perspective of KRaft, so I think it is okay to block even the
> > initialization of QuorumController until the cluster id is discovered.
> Just
> > like with brokers, we only block for 1 successful Fetch/Fetch Snapshot
> loop
> > until this data is known. One detail is that for auto-joining observers
> in
> > kraft.version=1, they need to wait until they persist cluster id before
> > they try to join the voter set.
> >
> > RE JR9.1:
> > This can also mean the broker skipped formatting, and does not have a
> > cluster id. In this case, it will persist cluster id to meta.properties.
> >
> > The other case is when the broker has a cluster.id in meta.properties.
> In
> > this case, the broker cannot discover a different cluster id via a
> > ClusterIdRecord in FetchResponse. In fact, the broker will not be able to
> > successfully complete any KRaft RPCs against the leader. For the broker
> to
> > receive a non-error FetchResponse with metadata records (which would be
> the
> > only way to learn of a different ClusterIdRecord), the KRaft leader
> checks
> > that the request cluster id is absent, or that the request cluster id
> > matches its own (which is the cluster id in its
> > meta.properties/ClusterIdRecord if the invariant I mentioned in my
> previous
> > message is enforced properly). This case could happen when bootstrap
> > endpoints point to the wrong cluster during restart of a node. The logic
> > above would result in startup timing out and shutting down the node
> because
> > the local node is not able to participate in KRaft for another cluster.
> >
> > RE JR9.2: Yes, the broker's startup will eventually timeout and fail. The
> > broker won't have cluster.id in meta.properties, and the cluster cannot
> > send the broker a cluster id via ClusterIdRecord. The same would apply
> for
> > an observer controller. This is a misconfiguration in my opinion.
> >
> > On Tue, Mar 3, 2026 at 12:22 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> > > Hi, Kevin,
> > >
> > > Thanks for the reply.
> > >
> > > JR1. ClusterIdRecord:
> > > It would be better for ClusterId to have the type uuid. This will make
> it
> > > consistent with topicId in TopicRecord. If an existing string can't be
> > > converted to uuid, we can fail the node. This shouldn't happen for a
> well
> > > formatted cluster, right?
> > >
> > > JR6. Have you decided what to include in this KIP? If this KIP still
> > > requires the formatting for bootstrap controllers, what's described
> here
> > > can't happen.
> > >
> > > JR7. "After KIP-1286, kafka operators no longer need to format all
> nodes"
> > > KIP-1286 seems to be the wrong KIP?
> > >
> > > JR8. "The readers of cluster id initialized during startup can wait for
> > > both the above before being initialized."
> > > What are those readers? Are they ok to block?
> > >
> > > JR9. A couple more upgrade scenarios.
> > > JR9.1 If the MV has been bumped, after a broker starts up, it discovers
> > > that the clusterId in ClusterIdRecord doesn't match the one in
> > > meta.properties. Will the broker fail?
> > > JR9.2 If the MV hasn't been bumped, a new broker with the new version
> of
> > > the software is started without formatting, will it fail during
> startup?
> > >
> > > Jun
> > >
> > > On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <ke...@gmail.com>
> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thanks for the replies and questions.
> > > >
> > > > RE JR1: Updated the KIP with the record schema for ClusterIdRecord.
> One
> > > > thing I'm not sure about yet is whether or not the record field
> should
> > be
> > > > of UUID or String type. This is because kafka's quickstart docs refer
> > to
> > > > setting `--cluster-id` to a UUID in the storage tool. However, many
> > > places
> > > > in kafka broker/controller code (e.g. the raft client, broker
> lifecycle
> > > > manager, and even the formatter itself) only require this type to be
> a
> > > > String. Since not all Strings are valid UUIDs, making this record
> field
> > > of
> > > > type UUID might be too restrictive and complicate upgrading the MV
> for
> > > > existing clusters, since they might have a non-UUID cluster id
> string,
> > > but
> > > > need to write this record when upgrading to an MV that supports this
> > > > feature. Let me know what you think.
> > > >
> > > > RE JR2: Any controller node formatted with `--standalone,
> > > > --initial-controllers` or who is part of the static voter set defined
> > by
> > > > `controller.quorum.voters` can write the ClusterIdRecord by including
> > the
> > > > `--cluster-id` argument to `kafka-storage format`. However, if the MV
> > of
> > > > the cluster supports it, there is exactly one writer of this record
> to
> > > the
> > > > cluster metadata partition. The writer is the first active
> controller,
> > > who
> > > > writes this record alongside other bootstrap metadata records (e.g.
> > > > metadata version) during controller activation. At this point, we
> > already
> > > > depend on MV existing, since the active controller writes these
> > bootstrap
> > > > metadata records as a transaction if the MV supports it. I think
> > writing
> > > > the cluster id record would follow a similar pattern.
> > > >
> > > > RE JR3: When a node formats, it will write the meta.properties file.
> > > During
> > > > formatting, a node must resolve the MV it wants to format with, which
> > is
> > > > explained more in RE JR5. I need to think about this more, but I
> think
> > we
> > > > should keep `--cluster-id` as a required flag for invoking the format
> > > > command. If a broker/observer controller does not format,
> > meta.properties
> > > > is written without cluster id immediately after startup (i.e. where
> we
> > > read
> > > > it from disk now in KafkaRaftServer).
> > > >
> > > > RE JR4: Yeah, will do. In this context, when I say observers I'm
> > > referring
> > > > to any controllers who are not part of the KRaft voter set when they
> > > start
> > > > kafka, or any brokers. I will make this explicit in the KIP. From the
> > > > perspective of this feature and KRaft leader election, controller
> nodes
> > > who
> > > > format with `--no-initial-controllers`, controller nodes who are not
> > part
> > > > of `controller.quorum.voters`, and brokers, all do not "need" to
> > format,
> > > > since they cannot become the active controller. This means they can
> > > resolve
> > > > metadata like the cluster id after discovering the leader. We have a
> > > > similar pattern with how controller nodes who format with
> > > > `--no-initial-controllers` discover the kraft version of the cluster.
> > > >
> > > > RE JR5: If a node formats, it must resolve a metadata version with
> > which
> > > to
> > > > format. This comes from the `--release-version/--feature` flag and
> > > defaults
> > > > to the latest production MV. Therefore, when a node formats with a
> > > metadata
> > > > version that supports this feature, it will write the ClusterIdRecord
> > to
> > > > its `0-0/bootstrap.checkpoint`. If the node formats with a metadata
> > > version
> > > > that does not support this feature, it does not write ClusterIdRecord
> > to
> > > > its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is
> > assumed
> > > > that this node is part of a cluster whose MV supports this.
> Otherwise,
> > > this
> > > > is a misconfiguration and the node will fail to register with the
> > leader
> > > > since there is no way for it to persist cluster id to its
> > meta.properties
> > > > without formatting.
> > > >
> > > > Although I did not specify this yet on the KIP explicitly, after some
> > > > offline discussion I think it makes sense to enforce the following
> > > > invariant as part of the feature design: if the persisted metadata
> > > version
> > > > supports this feature, the ClusterId record must also be persisted.
> > This
> > > is
> > > > enforceable on the write-path for MV, which occurs at two points--
> > during
> > > > formatting and during feature upgrades. There is a similar pattern
> with
> > > > kraft.version, as it gets written to disk at the same two points.
> > > >
> > > > RE JR6: The main motivation for writing cluster id to meta.properties
> > as
> > > > well is because it can act as a projection of the cluster metadata
> > > > partition which essentially only exposes the cluster id to readers.
> For
> > > > example, the raft layer needs to be aware of the cluster id for its
> own
> > > RPC
> > > > handling/validation, but raft cannot read metadata records. There are
> > > many
> > > > readers of this cluster id value during the startup of the cluster.
> > > > Therefore, avoiding a read of the metadata partition to discover the
> > > value
> > > > of this metadata will prevent more complications of the startup code.
> > > >
> > > > Best,
> > > > Kevin Wu
> > > >
> > > >
> > > > On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <
> dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hi, Kevin,
> > > > >
> > > > > Thanks for the KIP. A few comments.
> > > > >
> > > > > JR1. ClusterIdRecord : Could you define the record format?
> > > > >
> > > > > JR2. "a new MetadataVersion that supports encoding/decoding this
> > > record.
> > > > > This means that during formatting, the bootstrap ClusterIdRecord is
> > > only
> > > > > written if the cluster is formatted with a MV that supports this
> > > > feature."
> > > > > Could you describe who writes the ClusterIdRecord? Is it the leader
> > > > > controller? Also, when is the record written? Do we guarantee that
> MV
> > > is
> > > > > available at that time?
> > > > >
> > > > > JR3. "meta.properties can be written during kafka broker/controller
> > > > startup
> > > > > if it doesn't exist already (from formatting)"
> > > > > Could you describe when meta.properties is written? Is MV available
> > at
> > > > that
> > > > > time?
> > > > >
> > > > > JR4. "Introduce a metadata record for cluster id + observers
> persist
> > > > > cluster id to meta.properties from metadata publishing pipeline"
> > > > > Could you clarify what observers are? Are they observer controllers
> > or
> > > > are
> > > > > they brokers (which are referred to as observers to the
> controller)?
> > > > >
> > > > > JR5. "Bootstrap controllers can add a mandatory “cluster id” record
> > > > during
> > > > > formatting"
> > > > > This sounds like adding a ClusterIdRecord is optional. If so, could
> > you
> > > > > describe when a record will be added and when a record will not be
> > > added?
> > > > >
> > > > > JR6. "However, kafka should still be able to handle the case where
> a
> > > > leader
> > > > > is elected without a cluster id in meta.properties , since KRaft
> does
> > > not
> > > > > need cluster.id  in order to elect a leader.
> > > > >           In this case, the active controller will write a cluster
> id
> > > > > record during the bootstrap metadata write."
> > > > > Hmm, earlier, the KIP says "Upon discovering the cluster ID for the
> > > first
> > > > > time, these nodes need to persist this to meta.properties". Why do
> we
> > > > need
> > > > > to introduce a separate place to write the cluster id to
> > > > meta.properties.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <ke...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Manually bumping this thread after finalizing a design.
> > > > > > KIP link:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > > >
> > > > > > Best,
> > > > > > Kevin Wu
> > > > > >
> > > > > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hello all,
> > > > > > >
> > > > > > > I would like to start a discussion on KIP-1262, which proposes
> > > > removing
> > > > > > > the formatting requirement for brokers and observer
> controllers.
> > > > > > Currently,
> > > > > > > I am considering two high-level designs, and would appreciate
> > > > community
> > > > > > > feedback on both approaches to decide on a final design.
> > > > > > >
> > > > > > > KIP link:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > > > >
> > > > > > > Best,
> > > > > > > Kevin Wu
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Jun Rao via dev <de...@kafka.apache.org>]

Hi, Kevin,

Thanks for the reply. The KIP looks good to me now.

Jun

On Tue, Mar 3, 2026 at 4:54 AM Kevin Wu <ke...@gmail.com> wrote:

> Hi Jun,
>
> Thanks for the reply.
>
> RE JR1: "If an existing string can't be converted to uuid, we can fail the
> node. This shouldn't happen for a well formatted cluster, right?"
> Currently, you can format a cluster with a non-UUID cluster ID string, and
> kafka considers this "well-formatted" (i.e. formatting code accepts String,
> server startup works, and clusterId is a String in-memory etc.). Our
> documentation references formatting with a UUID cluster id generated via
> `kafka-storage random-uuid`, but this is not a requirement in the code. If
> we make this record have a UUID to be consistent with TopicRecord, it is
> not clear to me what the MV upgrade path is for existing clusters who
> formatted `meta.properties` with a non-UUID String. We have to write a new
> UUID cluster id, which violates the invariant that the cluster id cannot
> change over the lifetime of a cluster.
>
> RE JR6: I plan on still requiring bootstrap controllers to format. This
> means we should not expect a leader to be elected who does not have a
> cluster id. Bootstrap controllers will fail when reading in meta.properties
> in KafkaRaftServer. I will remove this section.
>
> RE JR7: Apologies, I mixed up the numbers with another KIP.
>
> RE JR8:
> For brokers, the readers of cluster id during startup are the
> BrokerLifecycleManager, KafkaApis, DynamicTopicClusterQuotaPublisher, and
> endpointReadyFutures. It is okay to block startup on fetching the cluster
> id from KRaft, since we also block startup on broker lifecycle manager
> initial catch up future. Discovering the cluster id value for the first
> time would only require a single FetchSnapshot or a Fetch of the bootstrap
> metadata records.
>
> For controllers, the readers are endpointReadyFutures,
> QuorumController, ControllerApis, ControllerRegistrationManager, and
> DynamicTopicClusterQuotaPublisher. For bootstrap controllers, this blocking
> does not occur. For observers, they are essentially brokers from the
> perspective of KRaft, so I think it is okay to block even the
> initialization of QuorumController until the cluster id is discovered. Just
> like with brokers, we only block for 1 successful Fetch/Fetch Snapshot loop
> until this data is known. One detail is that for auto-joining observers in
> kraft.version=1, they need to wait until they persist cluster id before
> they try to join the voter set.
>
> RE JR9.1:
> This can also mean the broker skipped formatting, and does not have a
> cluster id. In this case, it will persist cluster id to meta.properties.
>
> The other case is when the broker has a cluster.id in meta.properties. In
> this case, the broker cannot discover a different cluster id via a
> ClusterIdRecord in FetchResponse. In fact, the broker will not be able to
> successfully complete any KRaft RPCs against the leader. For the broker to
> receive a non-error FetchResponse with metadata records (which would be the
> only way to learn of a different ClusterIdRecord), the KRaft leader checks
> that the request cluster id is absent, or that the request cluster id
> matches its own (which is the cluster id in its
> meta.properties/ClusterIdRecord if the invariant I mentioned in my previous
> message is enforced properly). This case could happen when bootstrap
> endpoints point to the wrong cluster during restart of a node. The logic
> above would result in startup timing out and shutting down the node because
> the local node is not able to participate in KRaft for another cluster.
>
> RE JR9.2: Yes, the broker's startup will eventually timeout and fail. The
> broker won't have cluster.id in meta.properties, and the cluster cannot
> send the broker a cluster id via ClusterIdRecord. The same would apply for
> an observer controller. This is a misconfiguration in my opinion.
>
> On Tue, Mar 3, 2026 at 12:22 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi, Kevin,
> >
> > Thanks for the reply.
> >
> > JR1. ClusterIdRecord:
> > It would be better for ClusterId to have the type uuid. This will make it
> > consistent with topicId in TopicRecord. If an existing string can't be
> > converted to uuid, we can fail the node. This shouldn't happen for a well
> > formatted cluster, right?
> >
> > JR6. Have you decided what to include in this KIP? If this KIP still
> > requires the formatting for bootstrap controllers, what's described here
> > can't happen.
> >
> > JR7. "After KIP-1286, kafka operators no longer need to format all nodes"
> > KIP-1286 seems to be the wrong KIP?
> >
> > JR8. "The readers of cluster id initialized during startup can wait for
> > both the above before being initialized."
> > What are those readers? Are they ok to block?
> >
> > JR9. A couple more upgrade scenarios.
> > JR9.1 If the MV has been bumped, after a broker starts up, it discovers
> > that the clusterId in ClusterIdRecord doesn't match the one in
> > meta.properties. Will the broker fail?
> > JR9.2 If the MV hasn't been bumped, a new broker with the new version of
> > the software is started without formatting, will it fail during startup?
> >
> > Jun
> >
> > On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <ke...@gmail.com> wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for the replies and questions.
> > >
> > > RE JR1: Updated the KIP with the record schema for ClusterIdRecord. One
> > > thing I'm not sure about yet is whether or not the record field should
> be
> > > of UUID or String type. This is because kafka's quickstart docs refer
> to
> > > setting `--cluster-id` to a UUID in the storage tool. However, many
> > places
> > > in kafka broker/controller code (e.g. the raft client, broker lifecycle
> > > manager, and even the formatter itself) only require this type to be a
> > > String. Since not all Strings are valid UUIDs, making this record field
> > of
> > > type UUID might be too restrictive and complicate upgrading the MV for
> > > existing clusters, since they might have a non-UUID cluster id string,
> > but
> > > need to write this record when upgrading to an MV that supports this
> > > feature. Let me know what you think.
> > >
> > > RE JR2: Any controller node formatted with `--standalone,
> > > --initial-controllers` or who is part of the static voter set defined
> by
> > > `controller.quorum.voters` can write the ClusterIdRecord by including
> the
> > > `--cluster-id` argument to `kafka-storage format`. However, if the MV
> of
> > > the cluster supports it, there is exactly one writer of this record to
> > the
> > > cluster metadata partition. The writer is the first active controller,
> > who
> > > writes this record alongside other bootstrap metadata records (e.g.
> > > metadata version) during controller activation. At this point, we
> already
> > > depend on MV existing, since the active controller writes these
> bootstrap
> > > metadata records as a transaction if the MV supports it. I think
> writing
> > > the cluster id record would follow a similar pattern.
> > >
> > > RE JR3: When a node formats, it will write the meta.properties file.
> > During
> > > formatting, a node must resolve the MV it wants to format with, which
> is
> > > explained more in RE JR5. I need to think about this more, but I think
> we
> > > should keep `--cluster-id` as a required flag for invoking the format
> > > command. If a broker/observer controller does not format,
> meta.properties
> > > is written without cluster id immediately after startup (i.e. where we
> > read
> > > it from disk now in KafkaRaftServer).
> > >
> > > RE JR4: Yeah, will do. In this context, when I say observers I'm
> > referring
> > > to any controllers who are not part of the KRaft voter set when they
> > start
> > > kafka, or any brokers. I will make this explicit in the KIP. From the
> > > perspective of this feature and KRaft leader election, controller nodes
> > who
> > > format with `--no-initial-controllers`, controller nodes who are not
> part
> > > of `controller.quorum.voters`, and brokers, all do not "need" to
> format,
> > > since they cannot become the active controller. This means they can
> > resolve
> > > metadata like the cluster id after discovering the leader. We have a
> > > similar pattern with how controller nodes who format with
> > > `--no-initial-controllers` discover the kraft version of the cluster.
> > >
> > > RE JR5: If a node formats, it must resolve a metadata version with
> which
> > to
> > > format. This comes from the `--release-version/--feature` flag and
> > defaults
> > > to the latest production MV. Therefore, when a node formats with a
> > metadata
> > > version that supports this feature, it will write the ClusterIdRecord
> to
> > > its `0-0/bootstrap.checkpoint`. If the node formats with a metadata
> > version
> > > that does not support this feature, it does not write ClusterIdRecord
> to
> > > its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is
> assumed
> > > that this node is part of a cluster whose MV supports this. Otherwise,
> > this
> > > is a misconfiguration and the node will fail to register with the
> leader
> > > since there is no way for it to persist cluster id to its
> meta.properties
> > > without formatting.
> > >
> > > Although I did not specify this yet on the KIP explicitly, after some
> > > offline discussion I think it makes sense to enforce the following
> > > invariant as part of the feature design: if the persisted metadata
> > version
> > > supports this feature, the ClusterId record must also be persisted.
> This
> > is
> > > enforceable on the write-path for MV, which occurs at two points--
> during
> > > formatting and during feature upgrades. There is a similar pattern with
> > > kraft.version, as it gets written to disk at the same two points.
> > >
> > > RE JR6: The main motivation for writing cluster id to meta.properties
> as
> > > well is because it can act as a projection of the cluster metadata
> > > partition which essentially only exposes the cluster id to readers. For
> > > example, the raft layer needs to be aware of the cluster id for its own
> > RPC
> > > handling/validation, but raft cannot read metadata records. There are
> > many
> > > readers of this cluster id value during the startup of the cluster.
> > > Therefore, avoiding a read of the metadata partition to discover the
> > value
> > > of this metadata will prevent more complications of the startup code.
> > >
> > > Best,
> > > Kevin Wu
> > >
> > >
> > > On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <de...@kafka.apache.org>
> > > wrote:
> > >
> > > > Hi, Kevin,
> > > >
> > > > Thanks for the KIP. A few comments.
> > > >
> > > > JR1. ClusterIdRecord : Could you define the record format?
> > > >
> > > > JR2. "a new MetadataVersion that supports encoding/decoding this
> > record.
> > > > This means that during formatting, the bootstrap ClusterIdRecord is
> > only
> > > > written if the cluster is formatted with a MV that supports this
> > > feature."
> > > > Could you describe who writes the ClusterIdRecord? Is it the leader
> > > > controller? Also, when is the record written? Do we guarantee that MV
> > is
> > > > available at that time?
> > > >
> > > > JR3. "meta.properties can be written during kafka broker/controller
> > > startup
> > > > if it doesn't exist already (from formatting)"
> > > > Could you describe when meta.properties is written? Is MV available
> at
> > > that
> > > > time?
> > > >
> > > > JR4. "Introduce a metadata record for cluster id + observers persist
> > > > cluster id to meta.properties from metadata publishing pipeline"
> > > > Could you clarify what observers are? Are they observer controllers
> or
> > > are
> > > > they brokers (which are referred to as observers to the controller)?
> > > >
> > > > JR5. "Bootstrap controllers can add a mandatory “cluster id” record
> > > during
> > > > formatting"
> > > > This sounds like adding a ClusterIdRecord is optional. If so, could
> you
> > > > describe when a record will be added and when a record will not be
> > added?
> > > >
> > > > JR6. "However, kafka should still be able to handle the case where a
> > > leader
> > > > is elected without a cluster id in meta.properties , since KRaft does
> > not
> > > > need cluster.id  in order to elect a leader.
> > > >           In this case, the active controller will write a cluster id
> > > > record during the bootstrap metadata write."
> > > > Hmm, earlier, the KIP says "Upon discovering the cluster ID for the
> > first
> > > > time, these nodes need to persist this to meta.properties". Why do we
> > > need
> > > > to introduce a separate place to write the cluster id to
> > > meta.properties.
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <ke...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Manually bumping this thread after finalizing a design.
> > > > > KIP link:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > >
> > > > > Best,
> > > > > Kevin Wu
> > > > >
> > > > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > I would like to start a discussion on KIP-1262, which proposes
> > > removing
> > > > > > the formatting requirement for brokers and observer controllers.
> > > > > Currently,
> > > > > > I am considering two high-level designs, and would appreciate
> > > community
> > > > > > feedback on both approaches to decide on a final design.
> > > > > >
> > > > > > KIP link:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > > >
> > > > > > Best,
> > > > > > Kevin Wu
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi Jun,

Thanks for the reply.

RE JR1: "If an existing string can't be converted to uuid, we can fail the
node. This shouldn't happen for a well formatted cluster, right?"
Currently, you can format a cluster with a non-UUID cluster ID string, and
kafka considers this "well-formatted" (i.e. formatting code accepts String,
server startup works, and clusterId is a String in-memory etc.). Our
documentation references formatting with a UUID cluster id generated via
`kafka-storage random-uuid`, but this is not a requirement in the code. If
we make this record have a UUID to be consistent with TopicRecord, it is
not clear to me what the MV upgrade path is for existing clusters who
formatted `meta.properties` with a non-UUID String. We have to write a new
UUID cluster id, which violates the invariant that the cluster id cannot
change over the lifetime of a cluster.

RE JR6: I plan on still requiring bootstrap controllers to format. This
means we should not expect a leader to be elected who does not have a
cluster id. Bootstrap controllers will fail when reading in meta.properties
in KafkaRaftServer. I will remove this section.

RE JR7: Apologies, I mixed up the numbers with another KIP.

RE JR8:
For brokers, the readers of cluster id during startup are the
BrokerLifecycleManager, KafkaApis, DynamicTopicClusterQuotaPublisher, and
endpointReadyFutures. It is okay to block startup on fetching the cluster
id from KRaft, since we also block startup on broker lifecycle manager
initial catch up future. Discovering the cluster id value for the first
time would only require a single FetchSnapshot or a Fetch of the bootstrap
metadata records.

For controllers, the readers are endpointReadyFutures,
QuorumController, ControllerApis, ControllerRegistrationManager, and
DynamicTopicClusterQuotaPublisher. For bootstrap controllers, this blocking
does not occur. For observers, they are essentially brokers from the
perspective of KRaft, so I think it is okay to block even the
initialization of QuorumController until the cluster id is discovered. Just
like with brokers, we only block for 1 successful Fetch/Fetch Snapshot loop
until this data is known. One detail is that for auto-joining observers in
kraft.version=1, they need to wait until they persist cluster id before
they try to join the voter set.

RE JR9.1:
This can also mean the broker skipped formatting, and does not have a
cluster id. In this case, it will persist cluster id to meta.properties.

The other case is when the broker has a cluster.id in meta.properties. In
this case, the broker cannot discover a different cluster id via a
ClusterIdRecord in FetchResponse. In fact, the broker will not be able to
successfully complete any KRaft RPCs against the leader. For the broker to
receive a non-error FetchResponse with metadata records (which would be the
only way to learn of a different ClusterIdRecord), the KRaft leader checks
that the request cluster id is absent, or that the request cluster id
matches its own (which is the cluster id in its
meta.properties/ClusterIdRecord if the invariant I mentioned in my previous
message is enforced properly). This case could happen when bootstrap
endpoints point to the wrong cluster during restart of a node. The logic
above would result in startup timing out and shutting down the node because
the local node is not able to participate in KRaft for another cluster.

RE JR9.2: Yes, the broker's startup will eventually timeout and fail. The
broker won't have cluster.id in meta.properties, and the cluster cannot
send the broker a cluster id via ClusterIdRecord. The same would apply for
an observer controller. This is a misconfiguration in my opinion.

On Tue, Mar 3, 2026 at 12:22 AM Jun Rao via dev <de...@kafka.apache.org>
wrote:

> Hi, Kevin,
>
> Thanks for the reply.
>
> JR1. ClusterIdRecord:
> It would be better for ClusterId to have the type uuid. This will make it
> consistent with topicId in TopicRecord. If an existing string can't be
> converted to uuid, we can fail the node. This shouldn't happen for a well
> formatted cluster, right?
>
> JR6. Have you decided what to include in this KIP? If this KIP still
> requires the formatting for bootstrap controllers, what's described here
> can't happen.
>
> JR7. "After KIP-1286, kafka operators no longer need to format all nodes"
> KIP-1286 seems to be the wrong KIP?
>
> JR8. "The readers of cluster id initialized during startup can wait for
> both the above before being initialized."
> What are those readers? Are they ok to block?
>
> JR9. A couple more upgrade scenarios.
> JR9.1 If the MV has been bumped, after a broker starts up, it discovers
> that the clusterId in ClusterIdRecord doesn't match the one in
> meta.properties. Will the broker fail?
> JR9.2 If the MV hasn't been bumped, a new broker with the new version of
> the software is started without formatting, will it fail during startup?
>
> Jun
>
> On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <ke...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the replies and questions.
> >
> > RE JR1: Updated the KIP with the record schema for ClusterIdRecord. One
> > thing I'm not sure about yet is whether or not the record field should be
> > of UUID or String type. This is because kafka's quickstart docs refer to
> > setting `--cluster-id` to a UUID in the storage tool. However, many
> places
> > in kafka broker/controller code (e.g. the raft client, broker lifecycle
> > manager, and even the formatter itself) only require this type to be a
> > String. Since not all Strings are valid UUIDs, making this record field
> of
> > type UUID might be too restrictive and complicate upgrading the MV for
> > existing clusters, since they might have a non-UUID cluster id string,
> but
> > need to write this record when upgrading to an MV that supports this
> > feature. Let me know what you think.
> >
> > RE JR2: Any controller node formatted with `--standalone,
> > --initial-controllers` or who is part of the static voter set defined by
> > `controller.quorum.voters` can write the ClusterIdRecord by including the
> > `--cluster-id` argument to `kafka-storage format`. However, if the MV of
> > the cluster supports it, there is exactly one writer of this record to
> the
> > cluster metadata partition. The writer is the first active controller,
> who
> > writes this record alongside other bootstrap metadata records (e.g.
> > metadata version) during controller activation. At this point, we already
> > depend on MV existing, since the active controller writes these bootstrap
> > metadata records as a transaction if the MV supports it. I think writing
> > the cluster id record would follow a similar pattern.
> >
> > RE JR3: When a node formats, it will write the meta.properties file.
> During
> > formatting, a node must resolve the MV it wants to format with, which is
> > explained more in RE JR5. I need to think about this more, but I think we
> > should keep `--cluster-id` as a required flag for invoking the format
> > command. If a broker/observer controller does not format, meta.properties
> > is written without cluster id immediately after startup (i.e. where we
> read
> > it from disk now in KafkaRaftServer).
> >
> > RE JR4: Yeah, will do. In this context, when I say observers I'm
> referring
> > to any controllers who are not part of the KRaft voter set when they
> start
> > kafka, or any brokers. I will make this explicit in the KIP. From the
> > perspective of this feature and KRaft leader election, controller nodes
> who
> > format with `--no-initial-controllers`, controller nodes who are not part
> > of `controller.quorum.voters`, and brokers, all do not "need" to format,
> > since they cannot become the active controller. This means they can
> resolve
> > metadata like the cluster id after discovering the leader. We have a
> > similar pattern with how controller nodes who format with
> > `--no-initial-controllers` discover the kraft version of the cluster.
> >
> > RE JR5: If a node formats, it must resolve a metadata version with which
> to
> > format. This comes from the `--release-version/--feature` flag and
> defaults
> > to the latest production MV. Therefore, when a node formats with a
> metadata
> > version that supports this feature, it will write the ClusterIdRecord to
> > its `0-0/bootstrap.checkpoint`. If the node formats with a metadata
> version
> > that does not support this feature, it does not write ClusterIdRecord to
> > its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is assumed
> > that this node is part of a cluster whose MV supports this. Otherwise,
> this
> > is a misconfiguration and the node will fail to register with the leader
> > since there is no way for it to persist cluster id to its meta.properties
> > without formatting.
> >
> > Although I did not specify this yet on the KIP explicitly, after some
> > offline discussion I think it makes sense to enforce the following
> > invariant as part of the feature design: if the persisted metadata
> version
> > supports this feature, the ClusterId record must also be persisted. This
> is
> > enforceable on the write-path for MV, which occurs at two points-- during
> > formatting and during feature upgrades. There is a similar pattern with
> > kraft.version, as it gets written to disk at the same two points.
> >
> > RE JR6: The main motivation for writing cluster id to meta.properties as
> > well is because it can act as a projection of the cluster metadata
> > partition which essentially only exposes the cluster id to readers. For
> > example, the raft layer needs to be aware of the cluster id for its own
> RPC
> > handling/validation, but raft cannot read metadata records. There are
> many
> > readers of this cluster id value during the startup of the cluster.
> > Therefore, avoiding a read of the metadata partition to discover the
> value
> > of this metadata will prevent more complications of the startup code.
> >
> > Best,
> > Kevin Wu
> >
> >
> > On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> > > Hi, Kevin,
> > >
> > > Thanks for the KIP. A few comments.
> > >
> > > JR1. ClusterIdRecord : Could you define the record format?
> > >
> > > JR2. "a new MetadataVersion that supports encoding/decoding this
> record.
> > > This means that during formatting, the bootstrap ClusterIdRecord is
> only
> > > written if the cluster is formatted with a MV that supports this
> > feature."
> > > Could you describe who writes the ClusterIdRecord? Is it the leader
> > > controller? Also, when is the record written? Do we guarantee that MV
> is
> > > available at that time?
> > >
> > > JR3. "meta.properties can be written during kafka broker/controller
> > startup
> > > if it doesn't exist already (from formatting)"
> > > Could you describe when meta.properties is written? Is MV available at
> > that
> > > time?
> > >
> > > JR4. "Introduce a metadata record for cluster id + observers persist
> > > cluster id to meta.properties from metadata publishing pipeline"
> > > Could you clarify what observers are? Are they observer controllers or
> > are
> > > they brokers (which are referred to as observers to the controller)?
> > >
> > > JR5. "Bootstrap controllers can add a mandatory “cluster id” record
> > during
> > > formatting"
> > > This sounds like adding a ClusterIdRecord is optional. If so, could you
> > > describe when a record will be added and when a record will not be
> added?
> > >
> > > JR6. "However, kafka should still be able to handle the case where a
> > leader
> > > is elected without a cluster id in meta.properties , since KRaft does
> not
> > > need cluster.id  in order to elect a leader.
> > >           In this case, the active controller will write a cluster id
> > > record during the bootstrap metadata write."
> > > Hmm, earlier, the KIP says "Upon discovering the cluster ID for the
> first
> > > time, these nodes need to persist this to meta.properties". Why do we
> > need
> > > to introduce a separate place to write the cluster id to
> > meta.properties.
> > >
> > > Jun
> > >
> > >
> > > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <ke...@gmail.com>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Manually bumping this thread after finalizing a design.
> > > > KIP link:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > >
> > > > Best,
> > > > Kevin Wu
> > > >
> > > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com>
> > wrote:
> > > >
> > > > > Hello all,
> > > > >
> > > > > I would like to start a discussion on KIP-1262, which proposes
> > removing
> > > > > the formatting requirement for brokers and observer controllers.
> > > > Currently,
> > > > > I am considering two high-level designs, and would appreciate
> > community
> > > > > feedback on both approaches to decide on a final design.
> > > > >
> > > > > KIP link:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > > >
> > > > > Best,
> > > > > Kevin Wu
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Jun Rao via dev <de...@kafka.apache.org>]

Hi, Kevin,

Thanks for the reply.

JR1. ClusterIdRecord:
It would be better for ClusterId to have the type uuid. This will make it
consistent with topicId in TopicRecord. If an existing string can't be
converted to uuid, we can fail the node. This shouldn't happen for a well
formatted cluster, right?

JR6. Have you decided what to include in this KIP? If this KIP still
requires the formatting for bootstrap controllers, what's described here
can't happen.

JR7. "After KIP-1286, kafka operators no longer need to format all nodes"
KIP-1286 seems to be the wrong KIP?

JR8. "The readers of cluster id initialized during startup can wait for
both the above before being initialized."
What are those readers? Are they ok to block?

JR9. A couple more upgrade scenarios.
JR9.1 If the MV has been bumped, after a broker starts up, it discovers
that the clusterId in ClusterIdRecord doesn't match the one in
meta.properties. Will the broker fail?
JR9.2 If the MV hasn't been bumped, a new broker with the new version of
the software is started without formatting, will it fail during startup?

Jun

On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <ke...@gmail.com> wrote:

> Hi Jun,
>
> Thanks for the replies and questions.
>
> RE JR1: Updated the KIP with the record schema for ClusterIdRecord. One
> thing I'm not sure about yet is whether or not the record field should be
> of UUID or String type. This is because kafka's quickstart docs refer to
> setting `--cluster-id` to a UUID in the storage tool. However, many places
> in kafka broker/controller code (e.g. the raft client, broker lifecycle
> manager, and even the formatter itself) only require this type to be a
> String. Since not all Strings are valid UUIDs, making this record field of
> type UUID might be too restrictive and complicate upgrading the MV for
> existing clusters, since they might have a non-UUID cluster id string, but
> need to write this record when upgrading to an MV that supports this
> feature. Let me know what you think.
>
> RE JR2: Any controller node formatted with `--standalone,
> --initial-controllers` or who is part of the static voter set defined by
> `controller.quorum.voters` can write the ClusterIdRecord by including the
> `--cluster-id` argument to `kafka-storage format`. However, if the MV of
> the cluster supports it, there is exactly one writer of this record to the
> cluster metadata partition. The writer is the first active controller, who
> writes this record alongside other bootstrap metadata records (e.g.
> metadata version) during controller activation. At this point, we already
> depend on MV existing, since the active controller writes these bootstrap
> metadata records as a transaction if the MV supports it. I think writing
> the cluster id record would follow a similar pattern.
>
> RE JR3: When a node formats, it will write the meta.properties file. During
> formatting, a node must resolve the MV it wants to format with, which is
> explained more in RE JR5. I need to think about this more, but I think we
> should keep `--cluster-id` as a required flag for invoking the format
> command. If a broker/observer controller does not format, meta.properties
> is written without cluster id immediately after startup (i.e. where we read
> it from disk now in KafkaRaftServer).
>
> RE JR4: Yeah, will do. In this context, when I say observers I'm referring
> to any controllers who are not part of the KRaft voter set when they start
> kafka, or any brokers. I will make this explicit in the KIP. From the
> perspective of this feature and KRaft leader election, controller nodes who
> format with `--no-initial-controllers`, controller nodes who are not part
> of `controller.quorum.voters`, and brokers, all do not "need" to format,
> since they cannot become the active controller. This means they can resolve
> metadata like the cluster id after discovering the leader. We have a
> similar pattern with how controller nodes who format with
> `--no-initial-controllers` discover the kraft version of the cluster.
>
> RE JR5: If a node formats, it must resolve a metadata version with which to
> format. This comes from the `--release-version/--feature` flag and defaults
> to the latest production MV. Therefore, when a node formats with a metadata
> version that supports this feature, it will write the ClusterIdRecord to
> its `0-0/bootstrap.checkpoint`. If the node formats with a metadata version
> that does not support this feature, it does not write ClusterIdRecord to
> its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is assumed
> that this node is part of a cluster whose MV supports this. Otherwise, this
> is a misconfiguration and the node will fail to register with the leader
> since there is no way for it to persist cluster id to its meta.properties
> without formatting.
>
> Although I did not specify this yet on the KIP explicitly, after some
> offline discussion I think it makes sense to enforce the following
> invariant as part of the feature design: if the persisted metadata version
> supports this feature, the ClusterId record must also be persisted. This is
> enforceable on the write-path for MV, which occurs at two points-- during
> formatting and during feature upgrades. There is a similar pattern with
> kraft.version, as it gets written to disk at the same two points.
>
> RE JR6: The main motivation for writing cluster id to meta.properties as
> well is because it can act as a projection of the cluster metadata
> partition which essentially only exposes the cluster id to readers. For
> example, the raft layer needs to be aware of the cluster id for its own RPC
> handling/validation, but raft cannot read metadata records. There are many
> readers of this cluster id value during the startup of the cluster.
> Therefore, avoiding a read of the metadata partition to discover the value
> of this metadata will prevent more complications of the startup code.
>
> Best,
> Kevin Wu
>
>
> On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi, Kevin,
> >
> > Thanks for the KIP. A few comments.
> >
> > JR1. ClusterIdRecord : Could you define the record format?
> >
> > JR2. "a new MetadataVersion that supports encoding/decoding this record.
> > This means that during formatting, the bootstrap ClusterIdRecord is only
> > written if the cluster is formatted with a MV that supports this
> feature."
> > Could you describe who writes the ClusterIdRecord? Is it the leader
> > controller? Also, when is the record written? Do we guarantee that MV is
> > available at that time?
> >
> > JR3. "meta.properties can be written during kafka broker/controller
> startup
> > if it doesn't exist already (from formatting)"
> > Could you describe when meta.properties is written? Is MV available at
> that
> > time?
> >
> > JR4. "Introduce a metadata record for cluster id + observers persist
> > cluster id to meta.properties from metadata publishing pipeline"
> > Could you clarify what observers are? Are they observer controllers or
> are
> > they brokers (which are referred to as observers to the controller)?
> >
> > JR5. "Bootstrap controllers can add a mandatory “cluster id” record
> during
> > formatting"
> > This sounds like adding a ClusterIdRecord is optional. If so, could you
> > describe when a record will be added and when a record will not be added?
> >
> > JR6. "However, kafka should still be able to handle the case where a
> leader
> > is elected without a cluster id in meta.properties , since KRaft does not
> > need cluster.id  in order to elect a leader.
> >           In this case, the active controller will write a cluster id
> > record during the bootstrap metadata write."
> > Hmm, earlier, the KIP says "Upon discovering the cluster ID for the first
> > time, these nodes need to persist this to meta.properties". Why do we
> need
> > to introduce a separate place to write the cluster id to
> meta.properties.
> >
> > Jun
> >
> >
> > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <ke...@gmail.com>
> wrote:
> >
> > > Hi all,
> > >
> > > Manually bumping this thread after finalizing a design.
> > > KIP link:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > >
> > > Best,
> > > Kevin Wu
> > >
> > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com>
> wrote:
> > >
> > > > Hello all,
> > > >
> > > > I would like to start a discussion on KIP-1262, which proposes
> removing
> > > > the formatting requirement for brokers and observer controllers.
> > > Currently,
> > > > I am considering two high-level designs, and would appreciate
> community
> > > > feedback on both approaches to decide on a final design.
> > > >
> > > > KIP link:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > > >
> > > > Best,
> > > > Kevin Wu
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi Jun,

Thanks for the replies and questions.

RE JR1: Updated the KIP with the record schema for ClusterIdRecord. One
thing I'm not sure about yet is whether or not the record field should be
of UUID or String type. This is because kafka's quickstart docs refer to
setting `--cluster-id` to a UUID in the storage tool. However, many places
in kafka broker/controller code (e.g. the raft client, broker lifecycle
manager, and even the formatter itself) only require this type to be a
String. Since not all Strings are valid UUIDs, making this record field of
type UUID might be too restrictive and complicate upgrading the MV for
existing clusters, since they might have a non-UUID cluster id string, but
need to write this record when upgrading to an MV that supports this
feature. Let me know what you think.

RE JR2: Any controller node formatted with `--standalone,
--initial-controllers` or who is part of the static voter set defined by
`controller.quorum.voters` can write the ClusterIdRecord by including the
`--cluster-id` argument to `kafka-storage format`. However, if the MV of
the cluster supports it, there is exactly one writer of this record to the
cluster metadata partition. The writer is the first active controller, who
writes this record alongside other bootstrap metadata records (e.g.
metadata version) during controller activation. At this point, we already
depend on MV existing, since the active controller writes these bootstrap
metadata records as a transaction if the MV supports it. I think writing
the cluster id record would follow a similar pattern.

RE JR3: When a node formats, it will write the meta.properties file. During
formatting, a node must resolve the MV it wants to format with, which is
explained more in RE JR5. I need to think about this more, but I think we
should keep `--cluster-id` as a required flag for invoking the format
command. If a broker/observer controller does not format, meta.properties
is written without cluster id immediately after startup (i.e. where we read
it from disk now in KafkaRaftServer).

RE JR4: Yeah, will do. In this context, when I say observers I'm referring
to any controllers who are not part of the KRaft voter set when they start
kafka, or any brokers. I will make this explicit in the KIP. From the
perspective of this feature and KRaft leader election, controller nodes who
format with `--no-initial-controllers`, controller nodes who are not part
of `controller.quorum.voters`, and brokers, all do not "need" to format,
since they cannot become the active controller. This means they can resolve
metadata like the cluster id after discovering the leader. We have a
similar pattern with how controller nodes who format with
`--no-initial-controllers` discover the kraft version of the cluster.

RE JR5: If a node formats, it must resolve a metadata version with which to
format. This comes from the `--release-version/--feature` flag and defaults
to the latest production MV. Therefore, when a node formats with a metadata
version that supports this feature, it will write the ClusterIdRecord to
its `0-0/bootstrap.checkpoint`. If the node formats with a metadata version
that does not support this feature, it does not write ClusterIdRecord to
its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is assumed
that this node is part of a cluster whose MV supports this. Otherwise, this
is a misconfiguration and the node will fail to register with the leader
since there is no way for it to persist cluster id to its meta.properties
without formatting.

Although I did not specify this yet on the KIP explicitly, after some
offline discussion I think it makes sense to enforce the following
invariant as part of the feature design: if the persisted metadata version
supports this feature, the ClusterId record must also be persisted. This is
enforceable on the write-path for MV, which occurs at two points-- during
formatting and during feature upgrades. There is a similar pattern with
kraft.version, as it gets written to disk at the same two points.

RE JR6: The main motivation for writing cluster id to meta.properties as
well is because it can act as a projection of the cluster metadata
partition which essentially only exposes the cluster id to readers. For
example, the raft layer needs to be aware of the cluster id for its own RPC
handling/validation, but raft cannot read metadata records. There are many
readers of this cluster id value during the startup of the cluster.
Therefore, avoiding a read of the metadata partition to discover the value
of this metadata will prevent more complications of the startup code.

Best,
Kevin Wu


On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <de...@kafka.apache.org>
wrote:

> Hi, Kevin,
>
> Thanks for the KIP. A few comments.
>
> JR1. ClusterIdRecord : Could you define the record format?
>
> JR2. "a new MetadataVersion that supports encoding/decoding this record.
> This means that during formatting, the bootstrap ClusterIdRecord is only
> written if the cluster is formatted with a MV that supports this feature."
> Could you describe who writes the ClusterIdRecord? Is it the leader
> controller? Also, when is the record written? Do we guarantee that MV is
> available at that time?
>
> JR3. "meta.properties can be written during kafka broker/controller startup
> if it doesn't exist already (from formatting)"
> Could you describe when meta.properties is written? Is MV available at that
> time?
>
> JR4. "Introduce a metadata record for cluster id + observers persist
> cluster id to meta.properties from metadata publishing pipeline"
> Could you clarify what observers are? Are they observer controllers or are
> they brokers (which are referred to as observers to the controller)?
>
> JR5. "Bootstrap controllers can add a mandatory “cluster id” record during
> formatting"
> This sounds like adding a ClusterIdRecord is optional. If so, could you
> describe when a record will be added and when a record will not be added?
>
> JR6. "However, kafka should still be able to handle the case where a leader
> is elected without a cluster id in meta.properties , since KRaft does not
> need cluster.id  in order to elect a leader.
>           In this case, the active controller will write a cluster id
> record during the bootstrap metadata write."
> Hmm, earlier, the KIP says "Upon discovering the cluster ID for the first
> time, these nodes need to persist this to meta.properties". Why do we need
> to introduce a separate place to write the cluster id to  meta.properties.
>
> Jun
>
>
> On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <ke...@gmail.com> wrote:
>
> > Hi all,
> >
> > Manually bumping this thread after finalizing a design.
> > KIP link:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> >
> > Best,
> > Kevin Wu
> >
> > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com> wrote:
> >
> > > Hello all,
> > >
> > > I would like to start a discussion on KIP-1262, which proposes removing
> > > the formatting requirement for brokers and observer controllers.
> > Currently,
> > > I am considering two high-level designs, and would appreciate community
> > > feedback on both approaches to decide on a final design.
> > >
> > > KIP link:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> > >
> > > Best,
> > > Kevin Wu
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Jun Rao via dev <de...@kafka.apache.org>]

Hi, Kevin,

Thanks for the KIP. A few comments.

JR1. ClusterIdRecord : Could you define the record format?

JR2. "a new MetadataVersion that supports encoding/decoding this record.
This means that during formatting, the bootstrap ClusterIdRecord is only
written if the cluster is formatted with a MV that supports this feature."
Could you describe who writes the ClusterIdRecord? Is it the leader
controller? Also, when is the record written? Do we guarantee that MV is
available at that time?

JR3. "meta.properties can be written during kafka broker/controller startup
if it doesn't exist already (from formatting)"
Could you describe when meta.properties is written? Is MV available at that
time?

JR4. "Introduce a metadata record for cluster id + observers persist
cluster id to meta.properties from metadata publishing pipeline"
Could you clarify what observers are? Are they observer controllers or are
they brokers (which are referred to as observers to the controller)?

JR5. "Bootstrap controllers can add a mandatory “cluster id” record during
formatting"
This sounds like adding a ClusterIdRecord is optional. If so, could you
describe when a record will be added and when a record will not be added?

JR6. "However, kafka should still be able to handle the case where a leader
is elected without a cluster id in meta.properties , since KRaft does not
need cluster.id  in order to elect a leader.
          In this case, the active controller will write a cluster id
record during the bootstrap metadata write."
Hmm, earlier, the KIP says "Upon discovering the cluster ID for the first
time, these nodes need to persist this to meta.properties". Why do we need
to introduce a separate place to write the cluster id to  meta.properties.

Jun


On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <ke...@gmail.com> wrote:

> Hi all,
>
> Manually bumping this thread after finalizing a design.
> KIP link:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
>
> Best,
> Kevin Wu
>
> On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com> wrote:
>
> > Hello all,
> >
> > I would like to start a discussion on KIP-1262, which proposes removing
> > the formatting requirement for brokers and observer controllers.
> Currently,
> > I am considering two high-level designs, and would appreciate community
> > feedback on both approaches to decide on a final design.
> >
> > KIP link:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
> >
> > Best,
> > Kevin Wu
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi all,

Manually bumping this thread after finalizing a design.
KIP link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories

Best,
Kevin Wu

On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <ke...@gmail.com> wrote:

> Hello all,
>
> I would like to start a discussion on KIP-1262, which proposes removing
> the formatting requirement for brokers and observer controllers. Currently,
> I am considering two high-level designs, and would appreciate community
> feedback on both approaches to decide on a final design.
>
> KIP link:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories
>
> Best,
> Kevin Wu
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi Alyssa,

Thanks for the replies and questions.

RE AH8: Yes, that is correct.

RE AH9: When I say "before starting kafka," I am referring to invoking the
entry point in `Kafka.scala`. I can reword this sentence to make it more
clear.

RE AH10: Yes, the broker will never send a broker registration request with
an empty cluster id. This is because all the in-memory readers of cluster
id on the broker can block until the cluster id is known (either from their
own meta.properties, which is assumed to be correct if it exists, or from
fetching the ClusterIdRecord).

RE AH11: I am planning to add a docs change as part of this KIP's
implementation to explain this.

Best,
Kevin Wu

On Fri, Apr 3, 2026 at 5:53 PM Alyssa Huang via dev <de...@kafka.apache.org>
wrote:

> Hey Kevin,
>
> Some questions about your revisions:
>
> AH8:
>
> > This means that during formatting, the bootstrap ClusterIdRecord is only
> > written if the node is formatted with a MV that supports this feature.
> When
> > a node runs kafka-storage format...
>
> I want to make sure I understand this properly. Formatting (i.e. operators
> calling kafka-storage format) is no longer necessary for the basic use case
> (no SCRAM, non default features, etc). And this sentence is just saying, if
> you still choose to explicitly format, then "the bootstrap ClusterIdRecord
> is only written if the node is formatted with a MV that supports this
> feature." And if you don't explicitly format the cluster, as long as the
> cluster's MV supports auto-formatting, the first elected KRaft leader will
> write the `ClusterIdRecord` if it does not yet exist in the metadata log.
>
> AH9:
>
> > *Remove the requirement of nodes to format before starting kafka*
>
> Might it make sense to reword this to *Remove the requirement of manually
> formatting nodes*? "Before starting Kafka" is still tripping me up - I can
> interpret it as "no need to format at all" or "can format after kafka
> starts".
>
> AH10:
>
> > the KRaft leader (clusterid = Y) must either receive a request without
> > clusterid, or a request whose clusterid is Y. The broker fulfills neither
> > of these conditions.
>
>  Are you saying the broker would never send a request w/ an empty
> clusterId? E.g. in their BrokerRegistrationRequests? Why will that be the
> case.
>
> AH11:
> It would be helpful to have a summary of what the provisioning and startup
> flow would look like with full auto-formatting w/ respect to controllers,
> observers, and brokers. Most of the details are in the KIP but just touched
> on in many different places.
>
> I like the new design though, thanks for the changes!
> Alyssa
>
>
>
> On Wed, Apr 1, 2026 at 10:58 AM Kevin Wu <ke...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Sounds good. I will make an explicit section to document when formatting
> is
> > still required. Thanks again for the feedback and questions.
> >
> > Best,
> > Kevin Wu
> >
> > On Wed, Apr 1, 2026 at 12:02 PM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> > > Hi, Kevin,
> > >
> > > Thanks for the explanation. I guess the special thing with dynamic
> > > deployment is the need to write the VotersRecord. We can keep the
> design
> > in
> > > the KIP. Could you document when formatting is still required in the
> KIP?
> > >
> > > Jun
> > >
> > > On Tue, Mar 31, 2026 at 2:49 PM Kevin Wu <ke...@gmail.com>
> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > RE JR12:
> > > > Dynamic quorums can technically not require formatting, but I fear it
> > > will
> > > > cause cluster misconfigurations more easily than the static quorum
> > case.
> > > > The main difference between the two quorum deployments is that static
> > > > quorum cannot recover from data loss on a voter, whereas dynamic
> quorum
> > > > can. Below is my reasoning for why maintaining this requirement for
> > > dynamic
> > > > clusters is better for Kafka operators given that formatting each
> > node's
> > > > disk is currently required.
> > > >
> > > > The KRaft voter set in both static and dynamic deployments is a piece
> > of
> > > > bootstrapping data that requires some orchestration to manage.
> > Currently,
> > > > the static voter set is managed by the supplier of a `.properties`
> file
> > > > which contains `controller.quorum.voters`. In the static quorum case,
> > the
> > > > requirements from kafka on an orchestration layer to safely manage
> the
> > > > value of `controller.quorum.voters` config is pretty straightforward:
> > > > supply the same value on all nodes all the time forever. I think it
> is
> > > more
> > > > obvious to operators, without needing to know too much about how
> KRaft
> > > > works, that having different values for `controller.quorum.voters` is
> > > > incorrect and unsafe.
> > > >
> > > > The dynamic voter set's contents is initially managed by the caller
> of
> > > > `kafka-storage format`, and then it is managed by KRaft itself. I
> will
> > > just
> > > > focus on what a standalone dynamic controller deployment would look
> > like
> > > > without formatting, but bootstrapping a dynamic quorum with multiple
> > > > controllers is unsafe for the same reasons. In order to remove the
> > > > formatting requirement for dynamic clusters, we could imagine having
> > > > something like `controller.quorum.standalone.enabled`. When that
> config
> > > is
> > > > defined during startup, kafka writes the bootstrapping VotersRecord
> and
> > > > KRaftVersion that would be done during formatting. However, the
> > > > requirements from kafka on an orchestration layer to safely manage
> this
> > > > "standalone" config are more complicated than the static quorum case.
> > > They
> > > > are also not obvious without prior knowledge of bootstrapping
> quorums.
> > > This
> > > > config should only be set on one node whenever the cluster has not
> been
> > > > "bootstrapped," and it cannot be present on any nodes during startup
> if
> > > the
> > > > cluster has already been "bootstrapped." If this config is present
> > after
> > > > the cluster has a voter set, it can result in multiple KRaft leaders
> > if a
> > > > node with the standalone config defined experiences data loss and
> tries
> > > to
> > > > restart.
> > > >
> > > > This "unsafeness" WRT dynamic quorum also applies to `kafka-storage
> > > > format,` but that CLI does not bootstrap with dynamic quorum by
> > default,
> > > as
> > > > the user has to specify one of `--standalone`,
> `--initial-controllers`,
> > > or
> > > > `--no-initial-controllers` when `controller.quorum.voters` is not
> > > defined.
> > > > Maybe that is a sufficient argument that it is okay to introduce the
> > > > `controller.quorum.standalone.enabled` static config as part of this
> > KIP.
> > > > Interested to know what you think about this.
> > > >
> > > > Best,
> > > > Kevin Wu
> > > >
> > > > On Tue, Mar 31, 2026 at 11:34 AM Jun Rao via dev <
> dev@kafka.apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hi, Kevin,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > JR12. https://kafka.apache.org/42/operations/kraft/ specifies two
> > > > > deployment methods for KRaft: static and dynamic. Are you saying
> that
> > > > > dynamic still requires formatting while static doesn't? Could you
> > > explain
> > > > > why there is difference? BTW, which method do we recommend?
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Mar 30, 2026 at 9:03 AM Kevin Wu <ke...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Thanks for the reply.
> > > > > >
> > > > > > RE JR10: Yes, I will update the KIP to reflect that.
> > > > > >
> > > > > > RE JR11: Yeah, I think it is fine to write V2.
> > > > > >
> > > > > > RE JR 12: Is this in reference to KIP-853: Dynamic Quorum
> > > > > Reconfiguration?
> > > > > > If so, see: "However, operators still have the option to format
> > nodes
> > > > to
> > > > > > set the MV, feature versions, scram credentials, or to properly
> > > > > provision a
> > > > > > kraft.version=1 cluster." In order to bootstrap any dynamic
> quorum
> > > > (i.e.
> > > > > > kraft.version=1) with an initial voter set, it is required to
> > format
> > > a
> > > > > > controller(s) with either `--standalone` or
> `--initial-controllers`
> > > so
> > > > > that
> > > > > > a KRaft VotersRecord is part of the 0-0.checkpoint. Formatting
> > > > > controllers
> > > > > > is still needed if you want to specify a non-default feature
> level
> > or
> > > > > > metadata version, and kraft.version=1 would be a "non-default"
> > KRaft
> > > > > > version (mainly because it is not correct without formatting,
> > > described
> > > > > > below).
> > > > > >
> > > > > > I'm not sure if removing this formatting requirement for new
> > KIP-853
> > > > > > clusters is in-scope for this KIP. The main issue with this is:
> How
> > > > does
> > > > > a
> > > > > > node know it can safely write a "bootstrapping" 0-0.checkpoint
> with
> > > the
> > > > > > KRaft VotersRecord on startup of the kafka process without
> knowing
> > > any
> > > > > > state of the cluster? This can lead to split-brain when a node
> > writes
> > > > > this
> > > > > > for a cluster who has already elected a leader. Currently, the
> > caller
> > > > of
> > > > > > the kafka-storage format command is responsible for writing this
> > > > exactly
> > > > > > once for the lifetime of the cluster.
> > > > > >
> > > > > > Operators still have the option of starting kafka without
> > formatting,
> > > > and
> > > > > > then upgrading the kraft version to kraft.version=1. This path
> > allows
> > > > > for a
> > > > > > dynamic quorum without formatting the cluster.
> > > > > >
> > > > > > Thanks,
> > > > > > Kevin Wu
> > > > > >
> > > > > > On Fri, Mar 27, 2026 at 4:20 PM Jun Rao via dev <
> > > dev@kafka.apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi, Kevin,
> > > > > > >
> > > > > > > Thanks for the updated KIP. It's better if we can remove the
> > > > formatting
> > > > > > > requirements for all nodes.
> > > > > > >
> > > > > > > JR10. "The reason for this KIP is to remove the requirement of
> > > > brokers
> > > > > > > needing to run kafka-storage format  before starting Kafka."
> > > > > > > Should we change brokers to nodes?
> > > > > > >
> > > > > > > JR11. "When --cluster-id  is specified, the formatter writes
> > > > > > > meta.properties  V1."
> > > > > > > It's a bit weird for the new code to write in V1 format. Could
> it
> > > > write
> > > > > > in
> > > > > > > V2 format?
> > > > > > >
> > > > > > > JR12. Without formatting, is it true that one can only
> bootstrap
> > a
> > > > > > > standalone controller? In other words, does bootstrapping with
> > > > multiple
> > > > > > > controllers still require formatting?
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Thu, Mar 19, 2026 at 1:39 AM Kevin Wu <
> kevin.wu2412@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi José,
> > > > > > > >
> > > > > > > > Thanks for the replies and questions.
> > > > > > > >
> > > > > > > > RE JS1: "Can you clarify that this KIP removes the need for
> all
> > > > Kafka
> > > > > > > nodes
> > > > > > > > to be formatted prior to starting Kafka." Hmmm, I guess in
> the
> > > > static
> > > > > > > > cluster case that skips formatting having a newer software
> > > version
> > > > +
> > > > > > > older
> > > > > > > > MV is not a possible case, so I will remove that mention from
> > the
> > > > > KIP.
> > > > > > We
> > > > > > > > should default to the latest MV if we skip formatting, which
> > will
> > > > > > support
> > > > > > > > writing a ClusterIdRecord.
> > > > > > > >
> > > > > > > > Right now, it is not completely clear to me how we can allow
> > > > > bootstrap
> > > > > > > > controllers (this applies mainly for kraft.version=0, since
> > > > > > > kraft.version=1
> > > > > > > > cannot elect a leader without proper formatting) to also skip
> > > > > > formatting.
> > > > > > > > That is why I said in the proposed changes: "*Remove the
> > > > requirement
> > > > > of
> > > > > > > > brokers and observer controllers to format before starting
> > > > kafka"*. I
> > > > > > > agree
> > > > > > > > that KRaft can still elect a leader without clusterId in this
> > > case,
> > > > > but
> > > > > > > I'm
> > > > > > > > not completely sure how a QuorumController with an "empty"
> > > > clusterId
> > > > > > > which
> > > > > > > > needs to be set later, should behave. My working idea is
> > detailed
> > > > in
> > > > > RE
> > > > > > > > JS6. This is required because the active controller needs to
> > > > > generate a
> > > > > > > > clusterId and write it back to KRaft upon activation in order
> > for
> > > > the
> > > > > > > > committed `ClusterIdRecord` to appear in records passed to
> > > > > > > > `RaftListener#handleCommit()`, so we cannot block its
> > > > initialization.
> > > > > > > > Keeping the assumption that QuorumController.clusterId is
> final
> > > and
> > > > > > > > non-null would be nice, but that requires all KRaft voters to
> > > > format
> > > > > > > with a
> > > > > > > > cluster.id. Let me know what you think about the best way to
> > > > remove
> > > > > > this
> > > > > > > > requirement.
> > > > > > > >
> > > > > > > > RE JS2: My plan was to continue to write meta.properties V1
> > > during
> > > > > > > > formatting with a `cluster.id` field like today, but also
> > write
> > > a
> > > > > > > > `ClusterIdRecord` to the bootstrap snapshot for redundancy if
> > the
> > > > MV
> > > > > > > > supports it (I'm not sure if kafka is expected to handle only
> > > > partial
> > > > > > log
> > > > > > > > directory corruption/destruction). If the "bootstrap
> controller
> > > > > cluster
> > > > > > > id
> > > > > > > > check" from JS4 is correct, then the initial active
> controller
> > is
> > > > > > > > guaranteed to have a non-null `cluster.id` in
> meta.properties.
> > > So
> > > > > long
> > > > > > > as
> > > > > > > > the MV supports it, the active controller would then write
> > > > > > > ClusterIdRecord
> > > > > > > > as part of the bootstrap records.
> > > > > > > >
> > > > > > > > RE JS3: When I said this, I meant that the restriction of
> > waiting
> > > > for
> > > > > > the
> > > > > > > > discovery of cluster.id to persist it to meta.properties
> > during
> > > > > broker
> > > > > > > > startup is no more restrictive than what already currently
> > > exists,
> > > > > > which
> > > > > > > is
> > > > > > > > being caught up to the HWM in order to register with the
> active
> > > > > > > controller.
> > > > > > > >
> > > > > > > > RE JS 4: Yeah, I thought about this, specifically around the
> > > > > > > > kraft.version=1 case since it is less straightforward what a
> > > > > "bootstrap
> > > > > > > > controller" is. Under the current design, in kraft.version=0,
> > any
> > > > > node
> > > > > > > who
> > > > > > > > is part of the `controller.quorum.voters` config must have
> > > > > > > > `meta.properties` with `cluster.id`. In kraft.version=1, any
> > > node
> > > > > who
> > > > > > > has
> > > > > > > > a
> > > > > > > > `0-0.checkpoint` is considered a "bootstrap controller." This
> > is
> > > a
> > > > > > > > heuristic, but I believe it is correct, since in order for
> the
> > > > > > > > 0-0.checkpoint to not exist on a node which formatted with
> > > > > --standalone
> > > > > > > or
> > > > > > > > --initial-controllers, there must have either been another
> > > > checkpoint
> > > > > > > with
> > > > > > > > committed records, which imply an elected initial leader, or
> a
> > > disk
> > > > > > loss.
> > > > > > > > Whenever a voter with id X and initial directory-id A comes
> > back
> > > as
> > > > > (X,
> > > > > > > B),
> > > > > > > > this process incarnation is an observer from the perspective
> of
> > > > > KRaft,
> > > > > > > and
> > > > > > > > I think we can assume it has neither `meta.properties` or
> > > > > > > `0-0.checkpoint`
> > > > > > > > if the operator did not format it (assumption from RE JS2
> about
> > > the
> > > > > > kinds
> > > > > > > > of storage failures we expect to handle are not partial
> > directory
> > > > > > > > failures). In this case, the "bootstrap controller" check
> does
> > > not
> > > > > > apply
> > > > > > > to
> > > > > > > > (X, B), and if auto-join is enabled, it will follow the steps
> > > > > detailed
> > > > > > in
> > > > > > > > RE JS5 to recover and rejoin the voter set. If we remove the
> > > > > > requirement
> > > > > > > on
> > > > > > > > all nodes to format, then we would not need to implement
> these
> > > > > checks.
> > > > > > > >
> > > > > > > > RE JS5: An observer without clusterId who can auto-join will
> > > fetch
> > > > > > until
> > > > > > > > its KafkaRaftClient updates the cluster id in-memory
> > (basically,
> > > > > > > auto-join
> > > > > > > > is off until it discovers the leader's clusterId). If the
> > > observer
> > > > > has
> > > > > > > > clusterId, it needs to match the leader's to perform a
> > successful
> > > > > > fetch,
> > > > > > > > which is required for successfully adding a voter via
> > auto-join.
> > > > > > > >
> > > > > > > > RE JS6: Apologies, I meant to say a MetadataPublisher
> > registered
> > > to
> > > > > the
> > > > > > > > MetadataLoader. Although, looking at this again, maybe this
> > > > > discovery +
> > > > > > > > persistence of clusterId can be handled by a new RaftListener
> > > > > instead.
> > > > > > I
> > > > > > > > don't think we need the overhead of the MetadataImage +
> > > > MetadataDelta
> > > > > > for
> > > > > > > > this feature since a RaftListener's `handleCommit()` and
> > > > > > > > `handleLoadSnapshot()` contain `ClusterIdRecord`. However,
> this
> > > > means
> > > > > > > > needing a third listener besides the MetadataLoader and
> > > > > > > QuorumMetaListener,
> > > > > > > > and therefore an additional call to log#read() when handling
> > > KRaft
> > > > > > > commits
> > > > > > > > + snapshots. From my reading, it seems like the Kafka log
> layer
> > > > does
> > > > > > not
> > > > > > > > attempt any caching, and instead we rely on the OS page
> cache.
> > > > > Because
> > > > > > of
> > > > > > > > this, I think we should be using MetadataPublisher, but let
> me
> > > know
> > > > > > what
> > > > > > > > you think.
> > > > > > > >
> > > > > > > > I am thinking of using an AtomicReference<String> to
> represent
> > > the
> > > > > > > > clusterId in-memory. This RaftListener/MetadataPublisher will
> > be
> > > > the
> > > > > > only
> > > > > > > > writer to this value if it is not already defined by
> > > > meta.properties,
> > > > > > but
> > > > > > > > there are many readers of this value. The initial value of
> this
> > > > > > reference
> > > > > > > > is null or the cluster.id from meta.properties. Upon reading
> > > > > > > > `ClusterIdRecord`, the listener will throw an exception if it
> > > has a
> > > > > > > > non-null clusterId and reads a ClusterIdRecord with a
> different
> > > ID.
> > > > > If
> > > > > > it
> > > > > > > > does not have cluster.id set and reads a ClusterIdRecord, it
> > > will
> > > > > > update
> > > > > > > > the AtomicReference and persist cluster.id to
> meta.properties.
> > > Let
> > > > > me
> > > > > > > know
> > > > > > > > if this approach sounds reasonable to you.
> > > > > > > >
> > > > > > > > RE JS7: From what I understand about MetaPropertiesEnsemble
> and
> > > its
> > > > > > > > verify() method, I think it is reasonable to say our
> > > > > > > > RaftListener/MetadataPublisher will know how many (if any)
> > > > > > > > `meta.properties` files it is responsible for persisting
> > > > cluster.id
> > > > > to
> > > > > > > > during the current process incarnation when it starts up.
> > > Currently
> > > > > we
> > > > > > > only
> > > > > > > > validate the MetaPropertiesEnsemble in two places: during
> > > > formatting,
> > > > > > and
> > > > > > > > during node startup. From what I understand, scenarios 1 and
> 2
> > > > should
> > > > > > > only
> > > > > > > > occur alongside a restart of the kafka process (to generate a
> > new
> > > > > > > > directory-id and/or update log.dirs), but please correct me
> if
> > > this
> > > > > > > > assumption is wrong. I'm not sure if scenario 3 is referring
> > to a
> > > > > > partial
> > > > > > > > write of a given meta.properties (i.e. it does not contain
> > > > > cluster.id
> > > > > > ),
> > > > > > > or
> > > > > > > > not writing the discovered cluster.id to all meta.properties
> > > files
> > > > > on
> > > > > > > the
> > > > > > > > node before a crash. If a meta.properties does not exist in a
> > > > > > > log/metadata
> > > > > > > > log directory during startup, we need to write a V2 one
> > without a
> > > > > > > > cluster.id,
> > > > > > > > but we would be aware of this. If we succeed writing
> > cluster.id
> > > to
> > > > > at
> > > > > > > > least
> > > > > > > > one meta.properties via the ClusterIdRecord, I believe it is
> > safe
> > > > to
> > > > > > > write
> > > > > > > > that same value to the other meta.properties upon restart if
> > they
> > > > > exist
> > > > > > > > because cluster.id does not change.
> > > > > > > >
> > > > > > > > I may have previously removed this from the KIP, but given
> this
> > > > > > > discussion,
> > > > > > > > I believe it is only safe to update the in-memory cluster.id
> > > only
> > > > > > after
> > > > > > > > writing this to all meta.properties on a node.
> > > > > > > >
> > > > > > > > RE JS8: Okay, maybe I will just rewrite the section. My point
> > was
> > > > to
> > > > > > say
> > > > > > > > something like: a node's discovery of the leader's committed
> > > > > > cluster.id
> > > > > > > > relies on the discovery of a HWM and our
> > > > > RaftListener/MetadataPublisher
> > > > > > > to
> > > > > > > > be registered with the raft client, and that we need to wait
> > for
> > > > > these
> > > > > > > > things before the startup logic in Controller/BrokerServer
> > > > executes.
> > > > > > > > However, if our listener does not see the ClusterIdRecord in
> > > > > > > `handleCommit`
> > > > > > > > or `handleLoadSnapshot`, it can't do anything meaningful, so
> it
> > > is
> > > > > more
> > > > > > > > accurate to say we need to wait until ClusterIdRecord is
> > > committed.
> > > > > > > >
> > > > > > > > On Thu, Mar 19, 2026 at 12:57 AM José Armando García Sancio
> via
> > > > dev <
> > > > > > > > dev@kafka.apache.org> wrote:
> > > > > > > >
> > > > > > > > > Hi Kevin, Thanks for the KIP and excuse my delay response.
> > > > > > > > >
> > > > > > > > > JS1: Can you clarify that this KIP removes the need for all
> > > Kafka
> > > > > > > > > nodes to be formatted pior to starting Kafka. However, this
> > > > doesn't
> > > > > > > > > prevent users from formatting their broker with a cluster
> ID
> > if
> > > > > they
> > > > > > > > > prefer. This is especially needed for Kafka nodes formatted
> > > for a
> > > > > > > > > cluster using an MV that doesn't support this feature.
> > > > > > > > >
> > > > > > > > > JS2: How are you planning to implement "kafka-storage
> format
> > > > > > > > > --clusterid YYY --standalone"? Is that going to behave like
> > it
> > > > does
> > > > > > > > > today by writing the cluster id to the meta.properties
> files?
> > > Or
> > > > > are
> > > > > > > > > you planning to write the cluster id using the
> > ClusterIdRecord
> > > to
> > > > > the
> > > > > > > > > bootstrap.checkpoint or 0-0.checkpoint (after KIP-1170)?
> > > > > > > > >
> > > > > > > > > JS3: In one of your replies you say "Discovering the
> cluster
> > id
> > > > > value
> > > > > > > > > for the first time would only require a single
> FetchSnapshot
> > > or a
> > > > > > > > > Fetch of the bootstrap metadata records." This is not
> > entirely
> > > > > > > > > accurate. The best we can say is that brokers need to catch
> > up
> > > to
> > > > > the
> > > > > > > > > HWM before they can send a registration requests to the
> > active
> > > > > > > > > controller or it can start a few internal component.
> However,
> > > the
> > > > > > > > > broker already had this requirement prior to this KIP, so
> it
> > is
> > > > not
> > > > > > > > > new.
> > > > > > > > >
> > > > > > > > > JS4: In the KIP you mention "if meta.properties does not
> > exist
> > > > and
> > > > > > the
> > > > > > > > > node is a bootstrap controller, throw a runtime exception."
> > Can
> > > > you
> > > > > > > > > explain how you plan to implement this? One important
> aspect
> > to
> > > > > > > > > consider is that in KRaft voters (controllers) are
> identified
> > > by
> > > > > the
> > > > > > > > > node ID and directory ID. A node can recover from a disk
> > > failure
> > > > by
> > > > > > > > > coming back with the same node ID but a different directory
> > ID.
> > > > In
> > > > > > > > > this case, the controller should auto-recover if the
> > auto-join
> > > > > > feature
> > > > > > > > > is enabled.
> > > > > > > > >
> > > > > > > > > JS5: In the KIP you mention "One detail here is that
> observer
> > > > > > > > > controllers with auto-join must wait until they have a
> > cluster
> > > id
> > > > > > > > > before trying to add or remove themselves." I understand
> the
> > > > reason
> > > > > > > > > for this requirement. If a node auto-joins the controller
> > > > cluster,
> > > > > > you
> > > > > > > > > must guarantee that it knows the cluster id in case it
> > becomes
> > > > the
> > > > > > > > > leader and needs to write the ClusterIDRecord. Can you
> > > elaborate
> > > > on
> > > > > > > > > your implementation plan?
> > > > > > > > >
> > > > > > > > > JS6: In the KIP you mention "This can be implemented as a
> > > > > > > > > MetadataPublisher that registers to the raft client
> alongside
> > > the
> > > > > > > > > MetadataLoader." Metadata publishers don't register with
> the
> > > > KRaft
> > > > > > > > > client. RaftClient.Listener register with the KRaft client.
> > > > > Metadata
> > > > > > > > > publisher register with the metadata loader instead.
> > > > > > > > >
> > > > > > > > > JS7: One complexity is that there is a meta.properties per
> > log
> > > > > > > > > directory and metadata log directory. This means that in
> the
> > > > stable
> > > > > > > > > case the cluster ID exists in all the meta.properties
> files.
> > > > > > > > > Unfortunately, this may not be the case for several
> reasons:
> > 1)
> > > > the
> > > > > > > > > disk was replaced, 2) a new disk was added, or 3) the write
> > > > > operation
> > > > > > > > > was only partially successful. How do you plan to handle
> this
> > > > case?
> > > > > > > > > Consider that the controller and the broker can run on the
> > same
> > > > JVM
> > > > > > > > > and use a log directory different from the metadata log
> > > > directory.
> > > > > > > > > Controllers only read and write to the metadata log
> > directory.
> > > > > > > > >
> > > > > > > > > JS8: In the KIP you mention "Learning of a HWM from the
> > leader,
> > > > > which
> > > > > > > > > the leader allows for because it will send valid fetch
> > > responses
> > > > > back
> > > > > > > > > to nodes that do not have a cluster id." One implementation
> > > > > > complexity
> > > > > > > > > is that KRaft can discover the HWM and send a handleCommit
> > > event
> > > > > > > > > without having fetched all data up to the HWM. What KRaft
> > > > > guarantees
> > > > > > > > > is that the active leader will not receive a
> > handleLeaderChange
> > > > > event
> > > > > > > > > until it has caught up to the leader's epoch. How do you
> plan
> > > to
> > > > > > > > > implement this?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > --
> > > > > > > > > -José
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Alyssa Huang via dev <de...@kafka.apache.org>]

Hey Kevin,

Some questions about your revisions:

AH8:

> This means that during formatting, the bootstrap ClusterIdRecord is only
> written if the node is formatted with a MV that supports this feature. When
> a node runs kafka-storage format...

I want to make sure I understand this properly. Formatting (i.e. operators
calling kafka-storage format) is no longer necessary for the basic use case
(no SCRAM, non default features, etc). And this sentence is just saying, if
you still choose to explicitly format, then "the bootstrap ClusterIdRecord
is only written if the node is formatted with a MV that supports this
feature." And if you don't explicitly format the cluster, as long as the
cluster's MV supports auto-formatting, the first elected KRaft leader will
write the `ClusterIdRecord` if it does not yet exist in the metadata log.

AH9:

> *Remove the requirement of nodes to format before starting kafka*

Might it make sense to reword this to *Remove the requirement of manually
formatting nodes*? "Before starting Kafka" is still tripping me up - I can
interpret it as "no need to format at all" or "can format after kafka
starts".

AH10:

> the KRaft leader (clusterid = Y) must either receive a request without
> clusterid, or a request whose clusterid is Y. The broker fulfills neither
> of these conditions.

 Are you saying the broker would never send a request w/ an empty
clusterId? E.g. in their BrokerRegistrationRequests? Why will that be the
case.

AH11:
It would be helpful to have a summary of what the provisioning and startup
flow would look like with full auto-formatting w/ respect to controllers,
observers, and brokers. Most of the details are in the KIP but just touched
on in many different places.

I like the new design though, thanks for the changes!
Alyssa



On Wed, Apr 1, 2026 at 10:58 AM Kevin Wu <ke...@gmail.com> wrote:

> Hi Jun,
>
> Sounds good. I will make an explicit section to document when formatting is
> still required. Thanks again for the feedback and questions.
>
> Best,
> Kevin Wu
>
> On Wed, Apr 1, 2026 at 12:02 PM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi, Kevin,
> >
> > Thanks for the explanation. I guess the special thing with dynamic
> > deployment is the need to write the VotersRecord. We can keep the design
> in
> > the KIP. Could you document when formatting is still required in the KIP?
> >
> > Jun
> >
> > On Tue, Mar 31, 2026 at 2:49 PM Kevin Wu <ke...@gmail.com> wrote:
> >
> > > Hi Jun,
> > >
> > > RE JR12:
> > > Dynamic quorums can technically not require formatting, but I fear it
> > will
> > > cause cluster misconfigurations more easily than the static quorum
> case.
> > > The main difference between the two quorum deployments is that static
> > > quorum cannot recover from data loss on a voter, whereas dynamic quorum
> > > can. Below is my reasoning for why maintaining this requirement for
> > dynamic
> > > clusters is better for Kafka operators given that formatting each
> node's
> > > disk is currently required.
> > >
> > > The KRaft voter set in both static and dynamic deployments is a piece
> of
> > > bootstrapping data that requires some orchestration to manage.
> Currently,
> > > the static voter set is managed by the supplier of a `.properties` file
> > > which contains `controller.quorum.voters`. In the static quorum case,
> the
> > > requirements from kafka on an orchestration layer to safely manage the
> > > value of `controller.quorum.voters` config is pretty straightforward:
> > > supply the same value on all nodes all the time forever. I think it is
> > more
> > > obvious to operators, without needing to know too much about how KRaft
> > > works, that having different values for `controller.quorum.voters` is
> > > incorrect and unsafe.
> > >
> > > The dynamic voter set's contents is initially managed by the caller of
> > > `kafka-storage format`, and then it is managed by KRaft itself. I will
> > just
> > > focus on what a standalone dynamic controller deployment would look
> like
> > > without formatting, but bootstrapping a dynamic quorum with multiple
> > > controllers is unsafe for the same reasons. In order to remove the
> > > formatting requirement for dynamic clusters, we could imagine having
> > > something like `controller.quorum.standalone.enabled`. When that config
> > is
> > > defined during startup, kafka writes the bootstrapping VotersRecord and
> > > KRaftVersion that would be done during formatting. However, the
> > > requirements from kafka on an orchestration layer to safely manage this
> > > "standalone" config are more complicated than the static quorum case.
> > They
> > > are also not obvious without prior knowledge of bootstrapping quorums.
> > This
> > > config should only be set on one node whenever the cluster has not been
> > > "bootstrapped," and it cannot be present on any nodes during startup if
> > the
> > > cluster has already been "bootstrapped." If this config is present
> after
> > > the cluster has a voter set, it can result in multiple KRaft leaders
> if a
> > > node with the standalone config defined experiences data loss and tries
> > to
> > > restart.
> > >
> > > This "unsafeness" WRT dynamic quorum also applies to `kafka-storage
> > > format,` but that CLI does not bootstrap with dynamic quorum by
> default,
> > as
> > > the user has to specify one of `--standalone`, `--initial-controllers`,
> > or
> > > `--no-initial-controllers` when `controller.quorum.voters` is not
> > defined.
> > > Maybe that is a sufficient argument that it is okay to introduce the
> > > `controller.quorum.standalone.enabled` static config as part of this
> KIP.
> > > Interested to know what you think about this.
> > >
> > > Best,
> > > Kevin Wu
> > >
> > > On Tue, Mar 31, 2026 at 11:34 AM Jun Rao via dev <dev@kafka.apache.org
> >
> > > wrote:
> > >
> > > > Hi, Kevin,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > JR12. https://kafka.apache.org/42/operations/kraft/ specifies two
> > > > deployment methods for KRaft: static and dynamic. Are you saying that
> > > > dynamic still requires formatting while static doesn't? Could you
> > explain
> > > > why there is difference? BTW, which method do we recommend?
> > > >
> > > > Jun
> > > >
> > > > On Mon, Mar 30, 2026 at 9:03 AM Kevin Wu <ke...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > RE JR10: Yes, I will update the KIP to reflect that.
> > > > >
> > > > > RE JR11: Yeah, I think it is fine to write V2.
> > > > >
> > > > > RE JR 12: Is this in reference to KIP-853: Dynamic Quorum
> > > > Reconfiguration?
> > > > > If so, see: "However, operators still have the option to format
> nodes
> > > to
> > > > > set the MV, feature versions, scram credentials, or to properly
> > > > provision a
> > > > > kraft.version=1 cluster." In order to bootstrap any dynamic quorum
> > > (i.e.
> > > > > kraft.version=1) with an initial voter set, it is required to
> format
> > a
> > > > > controller(s) with either `--standalone` or `--initial-controllers`
> > so
> > > > that
> > > > > a KRaft VotersRecord is part of the 0-0.checkpoint. Formatting
> > > > controllers
> > > > > is still needed if you want to specify a non-default feature level
> or
> > > > > metadata version, and kraft.version=1 would be a "non-default"
> KRaft
> > > > > version (mainly because it is not correct without formatting,
> > described
> > > > > below).
> > > > >
> > > > > I'm not sure if removing this formatting requirement for new
> KIP-853
> > > > > clusters is in-scope for this KIP. The main issue with this is: How
> > > does
> > > > a
> > > > > node know it can safely write a "bootstrapping" 0-0.checkpoint with
> > the
> > > > > KRaft VotersRecord on startup of the kafka process without knowing
> > any
> > > > > state of the cluster? This can lead to split-brain when a node
> writes
> > > > this
> > > > > for a cluster who has already elected a leader. Currently, the
> caller
> > > of
> > > > > the kafka-storage format command is responsible for writing this
> > > exactly
> > > > > once for the lifetime of the cluster.
> > > > >
> > > > > Operators still have the option of starting kafka without
> formatting,
> > > and
> > > > > then upgrading the kraft version to kraft.version=1. This path
> allows
> > > > for a
> > > > > dynamic quorum without formatting the cluster.
> > > > >
> > > > > Thanks,
> > > > > Kevin Wu
> > > > >
> > > > > On Fri, Mar 27, 2026 at 4:20 PM Jun Rao via dev <
> > dev@kafka.apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi, Kevin,
> > > > > >
> > > > > > Thanks for the updated KIP. It's better if we can remove the
> > > formatting
> > > > > > requirements for all nodes.
> > > > > >
> > > > > > JR10. "The reason for this KIP is to remove the requirement of
> > > brokers
> > > > > > needing to run kafka-storage format  before starting Kafka."
> > > > > > Should we change brokers to nodes?
> > > > > >
> > > > > > JR11. "When --cluster-id  is specified, the formatter writes
> > > > > > meta.properties  V1."
> > > > > > It's a bit weird for the new code to write in V1 format. Could it
> > > write
> > > > > in
> > > > > > V2 format?
> > > > > >
> > > > > > JR12. Without formatting, is it true that one can only bootstrap
> a
> > > > > > standalone controller? In other words, does bootstrapping with
> > > multiple
> > > > > > controllers still require formatting?
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, Mar 19, 2026 at 1:39 AM Kevin Wu <kevin.wu2412@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi José,
> > > > > > >
> > > > > > > Thanks for the replies and questions.
> > > > > > >
> > > > > > > RE JS1: "Can you clarify that this KIP removes the need for all
> > > Kafka
> > > > > > nodes
> > > > > > > to be formatted prior to starting Kafka." Hmmm, I guess in the
> > > static
> > > > > > > cluster case that skips formatting having a newer software
> > version
> > > +
> > > > > > older
> > > > > > > MV is not a possible case, so I will remove that mention from
> the
> > > > KIP.
> > > > > We
> > > > > > > should default to the latest MV if we skip formatting, which
> will
> > > > > support
> > > > > > > writing a ClusterIdRecord.
> > > > > > >
> > > > > > > Right now, it is not completely clear to me how we can allow
> > > > bootstrap
> > > > > > > controllers (this applies mainly for kraft.version=0, since
> > > > > > kraft.version=1
> > > > > > > cannot elect a leader without proper formatting) to also skip
> > > > > formatting.
> > > > > > > That is why I said in the proposed changes: "*Remove the
> > > requirement
> > > > of
> > > > > > > brokers and observer controllers to format before starting
> > > kafka"*. I
> > > > > > agree
> > > > > > > that KRaft can still elect a leader without clusterId in this
> > case,
> > > > but
> > > > > > I'm
> > > > > > > not completely sure how a QuorumController with an "empty"
> > > clusterId
> > > > > > which
> > > > > > > needs to be set later, should behave. My working idea is
> detailed
> > > in
> > > > RE
> > > > > > > JS6. This is required because the active controller needs to
> > > > generate a
> > > > > > > clusterId and write it back to KRaft upon activation in order
> for
> > > the
> > > > > > > committed `ClusterIdRecord` to appear in records passed to
> > > > > > > `RaftListener#handleCommit()`, so we cannot block its
> > > initialization.
> > > > > > > Keeping the assumption that QuorumController.clusterId is final
> > and
> > > > > > > non-null would be nice, but that requires all KRaft voters to
> > > format
> > > > > > with a
> > > > > > > cluster.id. Let me know what you think about the best way to
> > > remove
> > > > > this
> > > > > > > requirement.
> > > > > > >
> > > > > > > RE JS2: My plan was to continue to write meta.properties V1
> > during
> > > > > > > formatting with a `cluster.id` field like today, but also
> write
> > a
> > > > > > > `ClusterIdRecord` to the bootstrap snapshot for redundancy if
> the
> > > MV
> > > > > > > supports it (I'm not sure if kafka is expected to handle only
> > > partial
> > > > > log
> > > > > > > directory corruption/destruction). If the "bootstrap controller
> > > > cluster
> > > > > > id
> > > > > > > check" from JS4 is correct, then the initial active controller
> is
> > > > > > > guaranteed to have a non-null `cluster.id` in meta.properties.
> > So
> > > > long
> > > > > > as
> > > > > > > the MV supports it, the active controller would then write
> > > > > > ClusterIdRecord
> > > > > > > as part of the bootstrap records.
> > > > > > >
> > > > > > > RE JS3: When I said this, I meant that the restriction of
> waiting
> > > for
> > > > > the
> > > > > > > discovery of cluster.id to persist it to meta.properties
> during
> > > > broker
> > > > > > > startup is no more restrictive than what already currently
> > exists,
> > > > > which
> > > > > > is
> > > > > > > being caught up to the HWM in order to register with the active
> > > > > > controller.
> > > > > > >
> > > > > > > RE JS 4: Yeah, I thought about this, specifically around the
> > > > > > > kraft.version=1 case since it is less straightforward what a
> > > > "bootstrap
> > > > > > > controller" is. Under the current design, in kraft.version=0,
> any
> > > > node
> > > > > > who
> > > > > > > is part of the `controller.quorum.voters` config must have
> > > > > > > `meta.properties` with `cluster.id`. In kraft.version=1, any
> > node
> > > > who
> > > > > > has
> > > > > > > a
> > > > > > > `0-0.checkpoint` is considered a "bootstrap controller." This
> is
> > a
> > > > > > > heuristic, but I believe it is correct, since in order for the
> > > > > > > 0-0.checkpoint to not exist on a node which formatted with
> > > > --standalone
> > > > > > or
> > > > > > > --initial-controllers, there must have either been another
> > > checkpoint
> > > > > > with
> > > > > > > committed records, which imply an elected initial leader, or a
> > disk
> > > > > loss.
> > > > > > > Whenever a voter with id X and initial directory-id A comes
> back
> > as
> > > > (X,
> > > > > > B),
> > > > > > > this process incarnation is an observer from the perspective of
> > > > KRaft,
> > > > > > and
> > > > > > > I think we can assume it has neither `meta.properties` or
> > > > > > `0-0.checkpoint`
> > > > > > > if the operator did not format it (assumption from RE JS2 about
> > the
> > > > > kinds
> > > > > > > of storage failures we expect to handle are not partial
> directory
> > > > > > > failures). In this case, the "bootstrap controller" check does
> > not
> > > > > apply
> > > > > > to
> > > > > > > (X, B), and if auto-join is enabled, it will follow the steps
> > > > detailed
> > > > > in
> > > > > > > RE JS5 to recover and rejoin the voter set. If we remove the
> > > > > requirement
> > > > > > on
> > > > > > > all nodes to format, then we would not need to implement these
> > > > checks.
> > > > > > >
> > > > > > > RE JS5: An observer without clusterId who can auto-join will
> > fetch
> > > > > until
> > > > > > > its KafkaRaftClient updates the cluster id in-memory
> (basically,
> > > > > > auto-join
> > > > > > > is off until it discovers the leader's clusterId). If the
> > observer
> > > > has
> > > > > > > clusterId, it needs to match the leader's to perform a
> successful
> > > > > fetch,
> > > > > > > which is required for successfully adding a voter via
> auto-join.
> > > > > > >
> > > > > > > RE JS6: Apologies, I meant to say a MetadataPublisher
> registered
> > to
> > > > the
> > > > > > > MetadataLoader. Although, looking at this again, maybe this
> > > > discovery +
> > > > > > > persistence of clusterId can be handled by a new RaftListener
> > > > instead.
> > > > > I
> > > > > > > don't think we need the overhead of the MetadataImage +
> > > MetadataDelta
> > > > > for
> > > > > > > this feature since a RaftListener's `handleCommit()` and
> > > > > > > `handleLoadSnapshot()` contain `ClusterIdRecord`. However, this
> > > means
> > > > > > > needing a third listener besides the MetadataLoader and
> > > > > > QuorumMetaListener,
> > > > > > > and therefore an additional call to log#read() when handling
> > KRaft
> > > > > > commits
> > > > > > > + snapshots. From my reading, it seems like the Kafka log layer
> > > does
> > > > > not
> > > > > > > attempt any caching, and instead we rely on the OS page cache.
> > > > Because
> > > > > of
> > > > > > > this, I think we should be using MetadataPublisher, but let me
> > know
> > > > > what
> > > > > > > you think.
> > > > > > >
> > > > > > > I am thinking of using an AtomicReference<String> to represent
> > the
> > > > > > > clusterId in-memory. This RaftListener/MetadataPublisher will
> be
> > > the
> > > > > only
> > > > > > > writer to this value if it is not already defined by
> > > meta.properties,
> > > > > but
> > > > > > > there are many readers of this value. The initial value of this
> > > > > reference
> > > > > > > is null or the cluster.id from meta.properties. Upon reading
> > > > > > > `ClusterIdRecord`, the listener will throw an exception if it
> > has a
> > > > > > > non-null clusterId and reads a ClusterIdRecord with a different
> > ID.
> > > > If
> > > > > it
> > > > > > > does not have cluster.id set and reads a ClusterIdRecord, it
> > will
> > > > > update
> > > > > > > the AtomicReference and persist cluster.id to meta.properties.
> > Let
> > > > me
> > > > > > know
> > > > > > > if this approach sounds reasonable to you.
> > > > > > >
> > > > > > > RE JS7: From what I understand about MetaPropertiesEnsemble and
> > its
> > > > > > > verify() method, I think it is reasonable to say our
> > > > > > > RaftListener/MetadataPublisher will know how many (if any)
> > > > > > > `meta.properties` files it is responsible for persisting
> > > cluster.id
> > > > to
> > > > > > > during the current process incarnation when it starts up.
> > Currently
> > > > we
> > > > > > only
> > > > > > > validate the MetaPropertiesEnsemble in two places: during
> > > formatting,
> > > > > and
> > > > > > > during node startup. From what I understand, scenarios 1 and 2
> > > should
> > > > > > only
> > > > > > > occur alongside a restart of the kafka process (to generate a
> new
> > > > > > > directory-id and/or update log.dirs), but please correct me if
> > this
> > > > > > > assumption is wrong. I'm not sure if scenario 3 is referring
> to a
> > > > > partial
> > > > > > > write of a given meta.properties (i.e. it does not contain
> > > > cluster.id
> > > > > ),
> > > > > > or
> > > > > > > not writing the discovered cluster.id to all meta.properties
> > files
> > > > on
> > > > > > the
> > > > > > > node before a crash. If a meta.properties does not exist in a
> > > > > > log/metadata
> > > > > > > log directory during startup, we need to write a V2 one
> without a
> > > > > > > cluster.id,
> > > > > > > but we would be aware of this. If we succeed writing
> cluster.id
> > to
> > > > at
> > > > > > > least
> > > > > > > one meta.properties via the ClusterIdRecord, I believe it is
> safe
> > > to
> > > > > > write
> > > > > > > that same value to the other meta.properties upon restart if
> they
> > > > exist
> > > > > > > because cluster.id does not change.
> > > > > > >
> > > > > > > I may have previously removed this from the KIP, but given this
> > > > > > discussion,
> > > > > > > I believe it is only safe to update the in-memory cluster.id
> > only
> > > > > after
> > > > > > > writing this to all meta.properties on a node.
> > > > > > >
> > > > > > > RE JS8: Okay, maybe I will just rewrite the section. My point
> was
> > > to
> > > > > say
> > > > > > > something like: a node's discovery of the leader's committed
> > > > > cluster.id
> > > > > > > relies on the discovery of a HWM and our
> > > > RaftListener/MetadataPublisher
> > > > > > to
> > > > > > > be registered with the raft client, and that we need to wait
> for
> > > > these
> > > > > > > things before the startup logic in Controller/BrokerServer
> > > executes.
> > > > > > > However, if our listener does not see the ClusterIdRecord in
> > > > > > `handleCommit`
> > > > > > > or `handleLoadSnapshot`, it can't do anything meaningful, so it
> > is
> > > > more
> > > > > > > accurate to say we need to wait until ClusterIdRecord is
> > committed.
> > > > > > >
> > > > > > > On Thu, Mar 19, 2026 at 12:57 AM José Armando García Sancio via
> > > dev <
> > > > > > > dev@kafka.apache.org> wrote:
> > > > > > >
> > > > > > > > Hi Kevin, Thanks for the KIP and excuse my delay response.
> > > > > > > >
> > > > > > > > JS1: Can you clarify that this KIP removes the need for all
> > Kafka
> > > > > > > > nodes to be formatted pior to starting Kafka. However, this
> > > doesn't
> > > > > > > > prevent users from formatting their broker with a cluster ID
> if
> > > > they
> > > > > > > > prefer. This is especially needed for Kafka nodes formatted
> > for a
> > > > > > > > cluster using an MV that doesn't support this feature.
> > > > > > > >
> > > > > > > > JS2: How are you planning to implement "kafka-storage format
> > > > > > > > --clusterid YYY --standalone"? Is that going to behave like
> it
> > > does
> > > > > > > > today by writing the cluster id to the meta.properties files?
> > Or
> > > > are
> > > > > > > > you planning to write the cluster id using the
> ClusterIdRecord
> > to
> > > > the
> > > > > > > > bootstrap.checkpoint or 0-0.checkpoint (after KIP-1170)?
> > > > > > > >
> > > > > > > > JS3: In one of your replies you say "Discovering the cluster
> id
> > > > value
> > > > > > > > for the first time would only require a single FetchSnapshot
> > or a
> > > > > > > > Fetch of the bootstrap metadata records." This is not
> entirely
> > > > > > > > accurate. The best we can say is that brokers need to catch
> up
> > to
> > > > the
> > > > > > > > HWM before they can send a registration requests to the
> active
> > > > > > > > controller or it can start a few internal component. However,
> > the
> > > > > > > > broker already had this requirement prior to this KIP, so it
> is
> > > not
> > > > > > > > new.
> > > > > > > >
> > > > > > > > JS4: In the KIP you mention "if meta.properties does not
> exist
> > > and
> > > > > the
> > > > > > > > node is a bootstrap controller, throw a runtime exception."
> Can
> > > you
> > > > > > > > explain how you plan to implement this? One important aspect
> to
> > > > > > > > consider is that in KRaft voters (controllers) are identified
> > by
> > > > the
> > > > > > > > node ID and directory ID. A node can recover from a disk
> > failure
> > > by
> > > > > > > > coming back with the same node ID but a different directory
> ID.
> > > In
> > > > > > > > this case, the controller should auto-recover if the
> auto-join
> > > > > feature
> > > > > > > > is enabled.
> > > > > > > >
> > > > > > > > JS5: In the KIP you mention "One detail here is that observer
> > > > > > > > controllers with auto-join must wait until they have a
> cluster
> > id
> > > > > > > > before trying to add or remove themselves." I understand the
> > > reason
> > > > > > > > for this requirement. If a node auto-joins the controller
> > > cluster,
> > > > > you
> > > > > > > > must guarantee that it knows the cluster id in case it
> becomes
> > > the
> > > > > > > > leader and needs to write the ClusterIDRecord. Can you
> > elaborate
> > > on
> > > > > > > > your implementation plan?
> > > > > > > >
> > > > > > > > JS6: In the KIP you mention "This can be implemented as a
> > > > > > > > MetadataPublisher that registers to the raft client alongside
> > the
> > > > > > > > MetadataLoader." Metadata publishers don't register with the
> > > KRaft
> > > > > > > > client. RaftClient.Listener register with the KRaft client.
> > > > Metadata
> > > > > > > > publisher register with the metadata loader instead.
> > > > > > > >
> > > > > > > > JS7: One complexity is that there is a meta.properties per
> log
> > > > > > > > directory and metadata log directory. This means that in the
> > > stable
> > > > > > > > case the cluster ID exists in all the meta.properties files.
> > > > > > > > Unfortunately, this may not be the case for several reasons:
> 1)
> > > the
> > > > > > > > disk was replaced, 2) a new disk was added, or 3) the write
> > > > operation
> > > > > > > > was only partially successful. How do you plan to handle this
> > > case?
> > > > > > > > Consider that the controller and the broker can run on the
> same
> > > JVM
> > > > > > > > and use a log directory different from the metadata log
> > > directory.
> > > > > > > > Controllers only read and write to the metadata log
> directory.
> > > > > > > >
> > > > > > > > JS8: In the KIP you mention "Learning of a HWM from the
> leader,
> > > > which
> > > > > > > > the leader allows for because it will send valid fetch
> > responses
> > > > back
> > > > > > > > to nodes that do not have a cluster id." One implementation
> > > > > complexity
> > > > > > > > is that KRaft can discover the HWM and send a handleCommit
> > event
> > > > > > > > without having fetched all data up to the HWM. What KRaft
> > > > guarantees
> > > > > > > > is that the active leader will not receive a
> handleLeaderChange
> > > > event
> > > > > > > > until it has caught up to the leader's epoch. How do you plan
> > to
> > > > > > > > implement this?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > --
> > > > > > > > -José
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi Jun,

Sounds good. I will make an explicit section to document when formatting is
still required. Thanks again for the feedback and questions.

Best,
Kevin Wu

On Wed, Apr 1, 2026 at 12:02 PM Jun Rao via dev <de...@kafka.apache.org>
wrote:

> Hi, Kevin,
>
> Thanks for the explanation. I guess the special thing with dynamic
> deployment is the need to write the VotersRecord. We can keep the design in
> the KIP. Could you document when formatting is still required in the KIP?
>
> Jun
>
> On Tue, Mar 31, 2026 at 2:49 PM Kevin Wu <ke...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > RE JR12:
> > Dynamic quorums can technically not require formatting, but I fear it
> will
> > cause cluster misconfigurations more easily than the static quorum case.
> > The main difference between the two quorum deployments is that static
> > quorum cannot recover from data loss on a voter, whereas dynamic quorum
> > can. Below is my reasoning for why maintaining this requirement for
> dynamic
> > clusters is better for Kafka operators given that formatting each node's
> > disk is currently required.
> >
> > The KRaft voter set in both static and dynamic deployments is a piece of
> > bootstrapping data that requires some orchestration to manage. Currently,
> > the static voter set is managed by the supplier of a `.properties` file
> > which contains `controller.quorum.voters`. In the static quorum case, the
> > requirements from kafka on an orchestration layer to safely manage the
> > value of `controller.quorum.voters` config is pretty straightforward:
> > supply the same value on all nodes all the time forever. I think it is
> more
> > obvious to operators, without needing to know too much about how KRaft
> > works, that having different values for `controller.quorum.voters` is
> > incorrect and unsafe.
> >
> > The dynamic voter set's contents is initially managed by the caller of
> > `kafka-storage format`, and then it is managed by KRaft itself. I will
> just
> > focus on what a standalone dynamic controller deployment would look like
> > without formatting, but bootstrapping a dynamic quorum with multiple
> > controllers is unsafe for the same reasons. In order to remove the
> > formatting requirement for dynamic clusters, we could imagine having
> > something like `controller.quorum.standalone.enabled`. When that config
> is
> > defined during startup, kafka writes the bootstrapping VotersRecord and
> > KRaftVersion that would be done during formatting. However, the
> > requirements from kafka on an orchestration layer to safely manage this
> > "standalone" config are more complicated than the static quorum case.
> They
> > are also not obvious without prior knowledge of bootstrapping quorums.
> This
> > config should only be set on one node whenever the cluster has not been
> > "bootstrapped," and it cannot be present on any nodes during startup if
> the
> > cluster has already been "bootstrapped." If this config is present after
> > the cluster has a voter set, it can result in multiple KRaft leaders if a
> > node with the standalone config defined experiences data loss and tries
> to
> > restart.
> >
> > This "unsafeness" WRT dynamic quorum also applies to `kafka-storage
> > format,` but that CLI does not bootstrap with dynamic quorum by default,
> as
> > the user has to specify one of `--standalone`, `--initial-controllers`,
> or
> > `--no-initial-controllers` when `controller.quorum.voters` is not
> defined.
> > Maybe that is a sufficient argument that it is okay to introduce the
> > `controller.quorum.standalone.enabled` static config as part of this KIP.
> > Interested to know what you think about this.
> >
> > Best,
> > Kevin Wu
> >
> > On Tue, Mar 31, 2026 at 11:34 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> > > Hi, Kevin,
> > >
> > > Thanks for the reply.
> > >
> > > JR12. https://kafka.apache.org/42/operations/kraft/ specifies two
> > > deployment methods for KRaft: static and dynamic. Are you saying that
> > > dynamic still requires formatting while static doesn't? Could you
> explain
> > > why there is difference? BTW, which method do we recommend?
> > >
> > > Jun
> > >
> > > On Mon, Mar 30, 2026 at 9:03 AM Kevin Wu <ke...@gmail.com>
> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > RE JR10: Yes, I will update the KIP to reflect that.
> > > >
> > > > RE JR11: Yeah, I think it is fine to write V2.
> > > >
> > > > RE JR 12: Is this in reference to KIP-853: Dynamic Quorum
> > > Reconfiguration?
> > > > If so, see: "However, operators still have the option to format nodes
> > to
> > > > set the MV, feature versions, scram credentials, or to properly
> > > provision a
> > > > kraft.version=1 cluster." In order to bootstrap any dynamic quorum
> > (i.e.
> > > > kraft.version=1) with an initial voter set, it is required to format
> a
> > > > controller(s) with either `--standalone` or `--initial-controllers`
> so
> > > that
> > > > a KRaft VotersRecord is part of the 0-0.checkpoint. Formatting
> > > controllers
> > > > is still needed if you want to specify a non-default feature level or
> > > > metadata version, and kraft.version=1 would be a "non-default" KRaft
> > > > version (mainly because it is not correct without formatting,
> described
> > > > below).
> > > >
> > > > I'm not sure if removing this formatting requirement for new KIP-853
> > > > clusters is in-scope for this KIP. The main issue with this is: How
> > does
> > > a
> > > > node know it can safely write a "bootstrapping" 0-0.checkpoint with
> the
> > > > KRaft VotersRecord on startup of the kafka process without knowing
> any
> > > > state of the cluster? This can lead to split-brain when a node writes
> > > this
> > > > for a cluster who has already elected a leader. Currently, the caller
> > of
> > > > the kafka-storage format command is responsible for writing this
> > exactly
> > > > once for the lifetime of the cluster.
> > > >
> > > > Operators still have the option of starting kafka without formatting,
> > and
> > > > then upgrading the kraft version to kraft.version=1. This path allows
> > > for a
> > > > dynamic quorum without formatting the cluster.
> > > >
> > > > Thanks,
> > > > Kevin Wu
> > > >
> > > > On Fri, Mar 27, 2026 at 4:20 PM Jun Rao via dev <
> dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hi, Kevin,
> > > > >
> > > > > Thanks for the updated KIP. It's better if we can remove the
> > formatting
> > > > > requirements for all nodes.
> > > > >
> > > > > JR10. "The reason for this KIP is to remove the requirement of
> > brokers
> > > > > needing to run kafka-storage format  before starting Kafka."
> > > > > Should we change brokers to nodes?
> > > > >
> > > > > JR11. "When --cluster-id  is specified, the formatter writes
> > > > > meta.properties  V1."
> > > > > It's a bit weird for the new code to write in V1 format. Could it
> > write
> > > > in
> > > > > V2 format?
> > > > >
> > > > > JR12. Without formatting, is it true that one can only bootstrap a
> > > > > standalone controller? In other words, does bootstrapping with
> > multiple
> > > > > controllers still require formatting?
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Mar 19, 2026 at 1:39 AM Kevin Wu <ke...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi José,
> > > > > >
> > > > > > Thanks for the replies and questions.
> > > > > >
> > > > > > RE JS1: "Can you clarify that this KIP removes the need for all
> > Kafka
> > > > > nodes
> > > > > > to be formatted prior to starting Kafka." Hmmm, I guess in the
> > static
> > > > > > cluster case that skips formatting having a newer software
> version
> > +
> > > > > older
> > > > > > MV is not a possible case, so I will remove that mention from the
> > > KIP.
> > > > We
> > > > > > should default to the latest MV if we skip formatting, which will
> > > > support
> > > > > > writing a ClusterIdRecord.
> > > > > >
> > > > > > Right now, it is not completely clear to me how we can allow
> > > bootstrap
> > > > > > controllers (this applies mainly for kraft.version=0, since
> > > > > kraft.version=1
> > > > > > cannot elect a leader without proper formatting) to also skip
> > > > formatting.
> > > > > > That is why I said in the proposed changes: "*Remove the
> > requirement
> > > of
> > > > > > brokers and observer controllers to format before starting
> > kafka"*. I
> > > > > agree
> > > > > > that KRaft can still elect a leader without clusterId in this
> case,
> > > but
> > > > > I'm
> > > > > > not completely sure how a QuorumController with an "empty"
> > clusterId
> > > > > which
> > > > > > needs to be set later, should behave. My working idea is detailed
> > in
> > > RE
> > > > > > JS6. This is required because the active controller needs to
> > > generate a
> > > > > > clusterId and write it back to KRaft upon activation in order for
> > the
> > > > > > committed `ClusterIdRecord` to appear in records passed to
> > > > > > `RaftListener#handleCommit()`, so we cannot block its
> > initialization.
> > > > > > Keeping the assumption that QuorumController.clusterId is final
> and
> > > > > > non-null would be nice, but that requires all KRaft voters to
> > format
> > > > > with a
> > > > > > cluster.id. Let me know what you think about the best way to
> > remove
> > > > this
> > > > > > requirement.
> > > > > >
> > > > > > RE JS2: My plan was to continue to write meta.properties V1
> during
> > > > > > formatting with a `cluster.id` field like today, but also write
> a
> > > > > > `ClusterIdRecord` to the bootstrap snapshot for redundancy if the
> > MV
> > > > > > supports it (I'm not sure if kafka is expected to handle only
> > partial
> > > > log
> > > > > > directory corruption/destruction). If the "bootstrap controller
> > > cluster
> > > > > id
> > > > > > check" from JS4 is correct, then the initial active controller is
> > > > > > guaranteed to have a non-null `cluster.id` in meta.properties.
> So
> > > long
> > > > > as
> > > > > > the MV supports it, the active controller would then write
> > > > > ClusterIdRecord
> > > > > > as part of the bootstrap records.
> > > > > >
> > > > > > RE JS3: When I said this, I meant that the restriction of waiting
> > for
> > > > the
> > > > > > discovery of cluster.id to persist it to meta.properties during
> > > broker
> > > > > > startup is no more restrictive than what already currently
> exists,
> > > > which
> > > > > is
> > > > > > being caught up to the HWM in order to register with the active
> > > > > controller.
> > > > > >
> > > > > > RE JS 4: Yeah, I thought about this, specifically around the
> > > > > > kraft.version=1 case since it is less straightforward what a
> > > "bootstrap
> > > > > > controller" is. Under the current design, in kraft.version=0, any
> > > node
> > > > > who
> > > > > > is part of the `controller.quorum.voters` config must have
> > > > > > `meta.properties` with `cluster.id`. In kraft.version=1, any
> node
> > > who
> > > > > has
> > > > > > a
> > > > > > `0-0.checkpoint` is considered a "bootstrap controller." This is
> a
> > > > > > heuristic, but I believe it is correct, since in order for the
> > > > > > 0-0.checkpoint to not exist on a node which formatted with
> > > --standalone
> > > > > or
> > > > > > --initial-controllers, there must have either been another
> > checkpoint
> > > > > with
> > > > > > committed records, which imply an elected initial leader, or a
> disk
> > > > loss.
> > > > > > Whenever a voter with id X and initial directory-id A comes back
> as
> > > (X,
> > > > > B),
> > > > > > this process incarnation is an observer from the perspective of
> > > KRaft,
> > > > > and
> > > > > > I think we can assume it has neither `meta.properties` or
> > > > > `0-0.checkpoint`
> > > > > > if the operator did not format it (assumption from RE JS2 about
> the
> > > > kinds
> > > > > > of storage failures we expect to handle are not partial directory
> > > > > > failures). In this case, the "bootstrap controller" check does
> not
> > > > apply
> > > > > to
> > > > > > (X, B), and if auto-join is enabled, it will follow the steps
> > > detailed
> > > > in
> > > > > > RE JS5 to recover and rejoin the voter set. If we remove the
> > > > requirement
> > > > > on
> > > > > > all nodes to format, then we would not need to implement these
> > > checks.
> > > > > >
> > > > > > RE JS5: An observer without clusterId who can auto-join will
> fetch
> > > > until
> > > > > > its KafkaRaftClient updates the cluster id in-memory (basically,
> > > > > auto-join
> > > > > > is off until it discovers the leader's clusterId). If the
> observer
> > > has
> > > > > > clusterId, it needs to match the leader's to perform a successful
> > > > fetch,
> > > > > > which is required for successfully adding a voter via auto-join.
> > > > > >
> > > > > > RE JS6: Apologies, I meant to say a MetadataPublisher registered
> to
> > > the
> > > > > > MetadataLoader. Although, looking at this again, maybe this
> > > discovery +
> > > > > > persistence of clusterId can be handled by a new RaftListener
> > > instead.
> > > > I
> > > > > > don't think we need the overhead of the MetadataImage +
> > MetadataDelta
> > > > for
> > > > > > this feature since a RaftListener's `handleCommit()` and
> > > > > > `handleLoadSnapshot()` contain `ClusterIdRecord`. However, this
> > means
> > > > > > needing a third listener besides the MetadataLoader and
> > > > > QuorumMetaListener,
> > > > > > and therefore an additional call to log#read() when handling
> KRaft
> > > > > commits
> > > > > > + snapshots. From my reading, it seems like the Kafka log layer
> > does
> > > > not
> > > > > > attempt any caching, and instead we rely on the OS page cache.
> > > Because
> > > > of
> > > > > > this, I think we should be using MetadataPublisher, but let me
> know
> > > > what
> > > > > > you think.
> > > > > >
> > > > > > I am thinking of using an AtomicReference<String> to represent
> the
> > > > > > clusterId in-memory. This RaftListener/MetadataPublisher will be
> > the
> > > > only
> > > > > > writer to this value if it is not already defined by
> > meta.properties,
> > > > but
> > > > > > there are many readers of this value. The initial value of this
> > > > reference
> > > > > > is null or the cluster.id from meta.properties. Upon reading
> > > > > > `ClusterIdRecord`, the listener will throw an exception if it
> has a
> > > > > > non-null clusterId and reads a ClusterIdRecord with a different
> ID.
> > > If
> > > > it
> > > > > > does not have cluster.id set and reads a ClusterIdRecord, it
> will
> > > > update
> > > > > > the AtomicReference and persist cluster.id to meta.properties.
> Let
> > > me
> > > > > know
> > > > > > if this approach sounds reasonable to you.
> > > > > >
> > > > > > RE JS7: From what I understand about MetaPropertiesEnsemble and
> its
> > > > > > verify() method, I think it is reasonable to say our
> > > > > > RaftListener/MetadataPublisher will know how many (if any)
> > > > > > `meta.properties` files it is responsible for persisting
> > cluster.id
> > > to
> > > > > > during the current process incarnation when it starts up.
> Currently
> > > we
> > > > > only
> > > > > > validate the MetaPropertiesEnsemble in two places: during
> > formatting,
> > > > and
> > > > > > during node startup. From what I understand, scenarios 1 and 2
> > should
> > > > > only
> > > > > > occur alongside a restart of the kafka process (to generate a new
> > > > > > directory-id and/or update log.dirs), but please correct me if
> this
> > > > > > assumption is wrong. I'm not sure if scenario 3 is referring to a
> > > > partial
> > > > > > write of a given meta.properties (i.e. it does not contain
> > > cluster.id
> > > > ),
> > > > > or
> > > > > > not writing the discovered cluster.id to all meta.properties
> files
> > > on
> > > > > the
> > > > > > node before a crash. If a meta.properties does not exist in a
> > > > > log/metadata
> > > > > > log directory during startup, we need to write a V2 one without a
> > > > > > cluster.id,
> > > > > > but we would be aware of this. If we succeed writing cluster.id
> to
> > > at
> > > > > > least
> > > > > > one meta.properties via the ClusterIdRecord, I believe it is safe
> > to
> > > > > write
> > > > > > that same value to the other meta.properties upon restart if they
> > > exist
> > > > > > because cluster.id does not change.
> > > > > >
> > > > > > I may have previously removed this from the KIP, but given this
> > > > > discussion,
> > > > > > I believe it is only safe to update the in-memory cluster.id
> only
> > > > after
> > > > > > writing this to all meta.properties on a node.
> > > > > >
> > > > > > RE JS8: Okay, maybe I will just rewrite the section. My point was
> > to
> > > > say
> > > > > > something like: a node's discovery of the leader's committed
> > > > cluster.id
> > > > > > relies on the discovery of a HWM and our
> > > RaftListener/MetadataPublisher
> > > > > to
> > > > > > be registered with the raft client, and that we need to wait for
> > > these
> > > > > > things before the startup logic in Controller/BrokerServer
> > executes.
> > > > > > However, if our listener does not see the ClusterIdRecord in
> > > > > `handleCommit`
> > > > > > or `handleLoadSnapshot`, it can't do anything meaningful, so it
> is
> > > more
> > > > > > accurate to say we need to wait until ClusterIdRecord is
> committed.
> > > > > >
> > > > > > On Thu, Mar 19, 2026 at 12:57 AM José Armando García Sancio via
> > dev <
> > > > > > dev@kafka.apache.org> wrote:
> > > > > >
> > > > > > > Hi Kevin, Thanks for the KIP and excuse my delay response.
> > > > > > >
> > > > > > > JS1: Can you clarify that this KIP removes the need for all
> Kafka
> > > > > > > nodes to be formatted pior to starting Kafka. However, this
> > doesn't
> > > > > > > prevent users from formatting their broker with a cluster ID if
> > > they
> > > > > > > prefer. This is especially needed for Kafka nodes formatted
> for a
> > > > > > > cluster using an MV that doesn't support this feature.
> > > > > > >
> > > > > > > JS2: How are you planning to implement "kafka-storage format
> > > > > > > --clusterid YYY --standalone"? Is that going to behave like it
> > does
> > > > > > > today by writing the cluster id to the meta.properties files?
> Or
> > > are
> > > > > > > you planning to write the cluster id using the ClusterIdRecord
> to
> > > the
> > > > > > > bootstrap.checkpoint or 0-0.checkpoint (after KIP-1170)?
> > > > > > >
> > > > > > > JS3: In one of your replies you say "Discovering the cluster id
> > > value
> > > > > > > for the first time would only require a single FetchSnapshot
> or a
> > > > > > > Fetch of the bootstrap metadata records." This is not entirely
> > > > > > > accurate. The best we can say is that brokers need to catch up
> to
> > > the
> > > > > > > HWM before they can send a registration requests to the active
> > > > > > > controller or it can start a few internal component. However,
> the
> > > > > > > broker already had this requirement prior to this KIP, so it is
> > not
> > > > > > > new.
> > > > > > >
> > > > > > > JS4: In the KIP you mention "if meta.properties does not exist
> > and
> > > > the
> > > > > > > node is a bootstrap controller, throw a runtime exception." Can
> > you
> > > > > > > explain how you plan to implement this? One important aspect to
> > > > > > > consider is that in KRaft voters (controllers) are identified
> by
> > > the
> > > > > > > node ID and directory ID. A node can recover from a disk
> failure
> > by
> > > > > > > coming back with the same node ID but a different directory ID.
> > In
> > > > > > > this case, the controller should auto-recover if the auto-join
> > > > feature
> > > > > > > is enabled.
> > > > > > >
> > > > > > > JS5: In the KIP you mention "One detail here is that observer
> > > > > > > controllers with auto-join must wait until they have a cluster
> id
> > > > > > > before trying to add or remove themselves." I understand the
> > reason
> > > > > > > for this requirement. If a node auto-joins the controller
> > cluster,
> > > > you
> > > > > > > must guarantee that it knows the cluster id in case it becomes
> > the
> > > > > > > leader and needs to write the ClusterIDRecord. Can you
> elaborate
> > on
> > > > > > > your implementation plan?
> > > > > > >
> > > > > > > JS6: In the KIP you mention "This can be implemented as a
> > > > > > > MetadataPublisher that registers to the raft client alongside
> the
> > > > > > > MetadataLoader." Metadata publishers don't register with the
> > KRaft
> > > > > > > client. RaftClient.Listener register with the KRaft client.
> > > Metadata
> > > > > > > publisher register with the metadata loader instead.
> > > > > > >
> > > > > > > JS7: One complexity is that there is a meta.properties per log
> > > > > > > directory and metadata log directory. This means that in the
> > stable
> > > > > > > case the cluster ID exists in all the meta.properties files.
> > > > > > > Unfortunately, this may not be the case for several reasons: 1)
> > the
> > > > > > > disk was replaced, 2) a new disk was added, or 3) the write
> > > operation
> > > > > > > was only partially successful. How do you plan to handle this
> > case?
> > > > > > > Consider that the controller and the broker can run on the same
> > JVM
> > > > > > > and use a log directory different from the metadata log
> > directory.
> > > > > > > Controllers only read and write to the metadata log directory.
> > > > > > >
> > > > > > > JS8: In the KIP you mention "Learning of a HWM from the leader,
> > > which
> > > > > > > the leader allows for because it will send valid fetch
> responses
> > > back
> > > > > > > to nodes that do not have a cluster id." One implementation
> > > > complexity
> > > > > > > is that KRaft can discover the HWM and send a handleCommit
> event
> > > > > > > without having fetched all data up to the HWM. What KRaft
> > > guarantees
> > > > > > > is that the active leader will not receive a handleLeaderChange
> > > event
> > > > > > > until it has caught up to the leader's epoch. How do you plan
> to
> > > > > > > implement this?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > --
> > > > > > > -José
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Jun Rao via dev <de...@kafka.apache.org>]

Hi, Kevin,

Thanks for the explanation. I guess the special thing with dynamic
deployment is the need to write the VotersRecord. We can keep the design in
the KIP. Could you document when formatting is still required in the KIP?

Jun

On Tue, Mar 31, 2026 at 2:49 PM Kevin Wu <ke...@gmail.com> wrote:

> Hi Jun,
>
> RE JR12:
> Dynamic quorums can technically not require formatting, but I fear it will
> cause cluster misconfigurations more easily than the static quorum case.
> The main difference between the two quorum deployments is that static
> quorum cannot recover from data loss on a voter, whereas dynamic quorum
> can. Below is my reasoning for why maintaining this requirement for dynamic
> clusters is better for Kafka operators given that formatting each node's
> disk is currently required.
>
> The KRaft voter set in both static and dynamic deployments is a piece of
> bootstrapping data that requires some orchestration to manage. Currently,
> the static voter set is managed by the supplier of a `.properties` file
> which contains `controller.quorum.voters`. In the static quorum case, the
> requirements from kafka on an orchestration layer to safely manage the
> value of `controller.quorum.voters` config is pretty straightforward:
> supply the same value on all nodes all the time forever. I think it is more
> obvious to operators, without needing to know too much about how KRaft
> works, that having different values for `controller.quorum.voters` is
> incorrect and unsafe.
>
> The dynamic voter set's contents is initially managed by the caller of
> `kafka-storage format`, and then it is managed by KRaft itself. I will just
> focus on what a standalone dynamic controller deployment would look like
> without formatting, but bootstrapping a dynamic quorum with multiple
> controllers is unsafe for the same reasons. In order to remove the
> formatting requirement for dynamic clusters, we could imagine having
> something like `controller.quorum.standalone.enabled`. When that config is
> defined during startup, kafka writes the bootstrapping VotersRecord and
> KRaftVersion that would be done during formatting. However, the
> requirements from kafka on an orchestration layer to safely manage this
> "standalone" config are more complicated than the static quorum case. They
> are also not obvious without prior knowledge of bootstrapping quorums. This
> config should only be set on one node whenever the cluster has not been
> "bootstrapped," and it cannot be present on any nodes during startup if the
> cluster has already been "bootstrapped." If this config is present after
> the cluster has a voter set, it can result in multiple KRaft leaders if a
> node with the standalone config defined experiences data loss and tries to
> restart.
>
> This "unsafeness" WRT dynamic quorum also applies to `kafka-storage
> format,` but that CLI does not bootstrap with dynamic quorum by default, as
> the user has to specify one of `--standalone`, `--initial-controllers`, or
> `--no-initial-controllers` when `controller.quorum.voters` is not defined.
> Maybe that is a sufficient argument that it is okay to introduce the
> `controller.quorum.standalone.enabled` static config as part of this KIP.
> Interested to know what you think about this.
>
> Best,
> Kevin Wu
>
> On Tue, Mar 31, 2026 at 11:34 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi, Kevin,
> >
> > Thanks for the reply.
> >
> > JR12. https://kafka.apache.org/42/operations/kraft/ specifies two
> > deployment methods for KRaft: static and dynamic. Are you saying that
> > dynamic still requires formatting while static doesn't? Could you explain
> > why there is difference? BTW, which method do we recommend?
> >
> > Jun
> >
> > On Mon, Mar 30, 2026 at 9:03 AM Kevin Wu <ke...@gmail.com> wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for the reply.
> > >
> > > RE JR10: Yes, I will update the KIP to reflect that.
> > >
> > > RE JR11: Yeah, I think it is fine to write V2.
> > >
> > > RE JR 12: Is this in reference to KIP-853: Dynamic Quorum
> > Reconfiguration?
> > > If so, see: "However, operators still have the option to format nodes
> to
> > > set the MV, feature versions, scram credentials, or to properly
> > provision a
> > > kraft.version=1 cluster." In order to bootstrap any dynamic quorum
> (i.e.
> > > kraft.version=1) with an initial voter set, it is required to format a
> > > controller(s) with either `--standalone` or `--initial-controllers` so
> > that
> > > a KRaft VotersRecord is part of the 0-0.checkpoint. Formatting
> > controllers
> > > is still needed if you want to specify a non-default feature level or
> > > metadata version, and kraft.version=1 would be a "non-default" KRaft
> > > version (mainly because it is not correct without formatting, described
> > > below).
> > >
> > > I'm not sure if removing this formatting requirement for new KIP-853
> > > clusters is in-scope for this KIP. The main issue with this is: How
> does
> > a
> > > node know it can safely write a "bootstrapping" 0-0.checkpoint with the
> > > KRaft VotersRecord on startup of the kafka process without knowing any
> > > state of the cluster? This can lead to split-brain when a node writes
> > this
> > > for a cluster who has already elected a leader. Currently, the caller
> of
> > > the kafka-storage format command is responsible for writing this
> exactly
> > > once for the lifetime of the cluster.
> > >
> > > Operators still have the option of starting kafka without formatting,
> and
> > > then upgrading the kraft version to kraft.version=1. This path allows
> > for a
> > > dynamic quorum without formatting the cluster.
> > >
> > > Thanks,
> > > Kevin Wu
> > >
> > > On Fri, Mar 27, 2026 at 4:20 PM Jun Rao via dev <de...@kafka.apache.org>
> > > wrote:
> > >
> > > > Hi, Kevin,
> > > >
> > > > Thanks for the updated KIP. It's better if we can remove the
> formatting
> > > > requirements for all nodes.
> > > >
> > > > JR10. "The reason for this KIP is to remove the requirement of
> brokers
> > > > needing to run kafka-storage format  before starting Kafka."
> > > > Should we change brokers to nodes?
> > > >
> > > > JR11. "When --cluster-id  is specified, the formatter writes
> > > > meta.properties  V1."
> > > > It's a bit weird for the new code to write in V1 format. Could it
> write
> > > in
> > > > V2 format?
> > > >
> > > > JR12. Without formatting, is it true that one can only bootstrap a
> > > > standalone controller? In other words, does bootstrapping with
> multiple
> > > > controllers still require formatting?
> > > >
> > > > Jun
> > > >
> > > > On Thu, Mar 19, 2026 at 1:39 AM Kevin Wu <ke...@gmail.com>
> > wrote:
> > > >
> > > > > Hi José,
> > > > >
> > > > > Thanks for the replies and questions.
> > > > >
> > > > > RE JS1: "Can you clarify that this KIP removes the need for all
> Kafka
> > > > nodes
> > > > > to be formatted prior to starting Kafka." Hmmm, I guess in the
> static
> > > > > cluster case that skips formatting having a newer software version
> +
> > > > older
> > > > > MV is not a possible case, so I will remove that mention from the
> > KIP.
> > > We
> > > > > should default to the latest MV if we skip formatting, which will
> > > support
> > > > > writing a ClusterIdRecord.
> > > > >
> > > > > Right now, it is not completely clear to me how we can allow
> > bootstrap
> > > > > controllers (this applies mainly for kraft.version=0, since
> > > > kraft.version=1
> > > > > cannot elect a leader without proper formatting) to also skip
> > > formatting.
> > > > > That is why I said in the proposed changes: "*Remove the
> requirement
> > of
> > > > > brokers and observer controllers to format before starting
> kafka"*. I
> > > > agree
> > > > > that KRaft can still elect a leader without clusterId in this case,
> > but
> > > > I'm
> > > > > not completely sure how a QuorumController with an "empty"
> clusterId
> > > > which
> > > > > needs to be set later, should behave. My working idea is detailed
> in
> > RE
> > > > > JS6. This is required because the active controller needs to
> > generate a
> > > > > clusterId and write it back to KRaft upon activation in order for
> the
> > > > > committed `ClusterIdRecord` to appear in records passed to
> > > > > `RaftListener#handleCommit()`, so we cannot block its
> initialization.
> > > > > Keeping the assumption that QuorumController.clusterId is final and
> > > > > non-null would be nice, but that requires all KRaft voters to
> format
> > > > with a
> > > > > cluster.id. Let me know what you think about the best way to
> remove
> > > this
> > > > > requirement.
> > > > >
> > > > > RE JS2: My plan was to continue to write meta.properties V1 during
> > > > > formatting with a `cluster.id` field like today, but also write a
> > > > > `ClusterIdRecord` to the bootstrap snapshot for redundancy if the
> MV
> > > > > supports it (I'm not sure if kafka is expected to handle only
> partial
> > > log
> > > > > directory corruption/destruction). If the "bootstrap controller
> > cluster
> > > > id
> > > > > check" from JS4 is correct, then the initial active controller is
> > > > > guaranteed to have a non-null `cluster.id` in meta.properties. So
> > long
> > > > as
> > > > > the MV supports it, the active controller would then write
> > > > ClusterIdRecord
> > > > > as part of the bootstrap records.
> > > > >
> > > > > RE JS3: When I said this, I meant that the restriction of waiting
> for
> > > the
> > > > > discovery of cluster.id to persist it to meta.properties during
> > broker
> > > > > startup is no more restrictive than what already currently exists,
> > > which
> > > > is
> > > > > being caught up to the HWM in order to register with the active
> > > > controller.
> > > > >
> > > > > RE JS 4: Yeah, I thought about this, specifically around the
> > > > > kraft.version=1 case since it is less straightforward what a
> > "bootstrap
> > > > > controller" is. Under the current design, in kraft.version=0, any
> > node
> > > > who
> > > > > is part of the `controller.quorum.voters` config must have
> > > > > `meta.properties` with `cluster.id`. In kraft.version=1, any node
> > who
> > > > has
> > > > > a
> > > > > `0-0.checkpoint` is considered a "bootstrap controller." This is a
> > > > > heuristic, but I believe it is correct, since in order for the
> > > > > 0-0.checkpoint to not exist on a node which formatted with
> > --standalone
> > > > or
> > > > > --initial-controllers, there must have either been another
> checkpoint
> > > > with
> > > > > committed records, which imply an elected initial leader, or a disk
> > > loss.
> > > > > Whenever a voter with id X and initial directory-id A comes back as
> > (X,
> > > > B),
> > > > > this process incarnation is an observer from the perspective of
> > KRaft,
> > > > and
> > > > > I think we can assume it has neither `meta.properties` or
> > > > `0-0.checkpoint`
> > > > > if the operator did not format it (assumption from RE JS2 about the
> > > kinds
> > > > > of storage failures we expect to handle are not partial directory
> > > > > failures). In this case, the "bootstrap controller" check does not
> > > apply
> > > > to
> > > > > (X, B), and if auto-join is enabled, it will follow the steps
> > detailed
> > > in
> > > > > RE JS5 to recover and rejoin the voter set. If we remove the
> > > requirement
> > > > on
> > > > > all nodes to format, then we would not need to implement these
> > checks.
> > > > >
> > > > > RE JS5: An observer without clusterId who can auto-join will fetch
> > > until
> > > > > its KafkaRaftClient updates the cluster id in-memory (basically,
> > > > auto-join
> > > > > is off until it discovers the leader's clusterId). If the observer
> > has
> > > > > clusterId, it needs to match the leader's to perform a successful
> > > fetch,
> > > > > which is required for successfully adding a voter via auto-join.
> > > > >
> > > > > RE JS6: Apologies, I meant to say a MetadataPublisher registered to
> > the
> > > > > MetadataLoader. Although, looking at this again, maybe this
> > discovery +
> > > > > persistence of clusterId can be handled by a new RaftListener
> > instead.
> > > I
> > > > > don't think we need the overhead of the MetadataImage +
> MetadataDelta
> > > for
> > > > > this feature since a RaftListener's `handleCommit()` and
> > > > > `handleLoadSnapshot()` contain `ClusterIdRecord`. However, this
> means
> > > > > needing a third listener besides the MetadataLoader and
> > > > QuorumMetaListener,
> > > > > and therefore an additional call to log#read() when handling KRaft
> > > > commits
> > > > > + snapshots. From my reading, it seems like the Kafka log layer
> does
> > > not
> > > > > attempt any caching, and instead we rely on the OS page cache.
> > Because
> > > of
> > > > > this, I think we should be using MetadataPublisher, but let me know
> > > what
> > > > > you think.
> > > > >
> > > > > I am thinking of using an AtomicReference<String> to represent the
> > > > > clusterId in-memory. This RaftListener/MetadataPublisher will be
> the
> > > only
> > > > > writer to this value if it is not already defined by
> meta.properties,
> > > but
> > > > > there are many readers of this value. The initial value of this
> > > reference
> > > > > is null or the cluster.id from meta.properties. Upon reading
> > > > > `ClusterIdRecord`, the listener will throw an exception if it has a
> > > > > non-null clusterId and reads a ClusterIdRecord with a different ID.
> > If
> > > it
> > > > > does not have cluster.id set and reads a ClusterIdRecord, it will
> > > update
> > > > > the AtomicReference and persist cluster.id to meta.properties. Let
> > me
> > > > know
> > > > > if this approach sounds reasonable to you.
> > > > >
> > > > > RE JS7: From what I understand about MetaPropertiesEnsemble and its
> > > > > verify() method, I think it is reasonable to say our
> > > > > RaftListener/MetadataPublisher will know how many (if any)
> > > > > `meta.properties` files it is responsible for persisting
> cluster.id
> > to
> > > > > during the current process incarnation when it starts up. Currently
> > we
> > > > only
> > > > > validate the MetaPropertiesEnsemble in two places: during
> formatting,
> > > and
> > > > > during node startup. From what I understand, scenarios 1 and 2
> should
> > > > only
> > > > > occur alongside a restart of the kafka process (to generate a new
> > > > > directory-id and/or update log.dirs), but please correct me if this
> > > > > assumption is wrong. I'm not sure if scenario 3 is referring to a
> > > partial
> > > > > write of a given meta.properties (i.e. it does not contain
> > cluster.id
> > > ),
> > > > or
> > > > > not writing the discovered cluster.id to all meta.properties files
> > on
> > > > the
> > > > > node before a crash. If a meta.properties does not exist in a
> > > > log/metadata
> > > > > log directory during startup, we need to write a V2 one without a
> > > > > cluster.id,
> > > > > but we would be aware of this. If we succeed writing cluster.id to
> > at
> > > > > least
> > > > > one meta.properties via the ClusterIdRecord, I believe it is safe
> to
> > > > write
> > > > > that same value to the other meta.properties upon restart if they
> > exist
> > > > > because cluster.id does not change.
> > > > >
> > > > > I may have previously removed this from the KIP, but given this
> > > > discussion,
> > > > > I believe it is only safe to update the in-memory cluster.id only
> > > after
> > > > > writing this to all meta.properties on a node.
> > > > >
> > > > > RE JS8: Okay, maybe I will just rewrite the section. My point was
> to
> > > say
> > > > > something like: a node's discovery of the leader's committed
> > > cluster.id
> > > > > relies on the discovery of a HWM and our
> > RaftListener/MetadataPublisher
> > > > to
> > > > > be registered with the raft client, and that we need to wait for
> > these
> > > > > things before the startup logic in Controller/BrokerServer
> executes.
> > > > > However, if our listener does not see the ClusterIdRecord in
> > > > `handleCommit`
> > > > > or `handleLoadSnapshot`, it can't do anything meaningful, so it is
> > more
> > > > > accurate to say we need to wait until ClusterIdRecord is committed.
> > > > >
> > > > > On Thu, Mar 19, 2026 at 12:57 AM José Armando García Sancio via
> dev <
> > > > > dev@kafka.apache.org> wrote:
> > > > >
> > > > > > Hi Kevin, Thanks for the KIP and excuse my delay response.
> > > > > >
> > > > > > JS1: Can you clarify that this KIP removes the need for all Kafka
> > > > > > nodes to be formatted pior to starting Kafka. However, this
> doesn't
> > > > > > prevent users from formatting their broker with a cluster ID if
> > they
> > > > > > prefer. This is especially needed for Kafka nodes formatted for a
> > > > > > cluster using an MV that doesn't support this feature.
> > > > > >
> > > > > > JS2: How are you planning to implement "kafka-storage format
> > > > > > --clusterid YYY --standalone"? Is that going to behave like it
> does
> > > > > > today by writing the cluster id to the meta.properties files? Or
> > are
> > > > > > you planning to write the cluster id using the ClusterIdRecord to
> > the
> > > > > > bootstrap.checkpoint or 0-0.checkpoint (after KIP-1170)?
> > > > > >
> > > > > > JS3: In one of your replies you say "Discovering the cluster id
> > value
> > > > > > for the first time would only require a single FetchSnapshot or a
> > > > > > Fetch of the bootstrap metadata records." This is not entirely
> > > > > > accurate. The best we can say is that brokers need to catch up to
> > the
> > > > > > HWM before they can send a registration requests to the active
> > > > > > controller or it can start a few internal component. However, the
> > > > > > broker already had this requirement prior to this KIP, so it is
> not
> > > > > > new.
> > > > > >
> > > > > > JS4: In the KIP you mention "if meta.properties does not exist
> and
> > > the
> > > > > > node is a bootstrap controller, throw a runtime exception." Can
> you
> > > > > > explain how you plan to implement this? One important aspect to
> > > > > > consider is that in KRaft voters (controllers) are identified by
> > the
> > > > > > node ID and directory ID. A node can recover from a disk failure
> by
> > > > > > coming back with the same node ID but a different directory ID.
> In
> > > > > > this case, the controller should auto-recover if the auto-join
> > > feature
> > > > > > is enabled.
> > > > > >
> > > > > > JS5: In the KIP you mention "One detail here is that observer
> > > > > > controllers with auto-join must wait until they have a cluster id
> > > > > > before trying to add or remove themselves." I understand the
> reason
> > > > > > for this requirement. If a node auto-joins the controller
> cluster,
> > > you
> > > > > > must guarantee that it knows the cluster id in case it becomes
> the
> > > > > > leader and needs to write the ClusterIDRecord. Can you elaborate
> on
> > > > > > your implementation plan?
> > > > > >
> > > > > > JS6: In the KIP you mention "This can be implemented as a
> > > > > > MetadataPublisher that registers to the raft client alongside the
> > > > > > MetadataLoader." Metadata publishers don't register with the
> KRaft
> > > > > > client. RaftClient.Listener register with the KRaft client.
> > Metadata
> > > > > > publisher register with the metadata loader instead.
> > > > > >
> > > > > > JS7: One complexity is that there is a meta.properties per log
> > > > > > directory and metadata log directory. This means that in the
> stable
> > > > > > case the cluster ID exists in all the meta.properties files.
> > > > > > Unfortunately, this may not be the case for several reasons: 1)
> the
> > > > > > disk was replaced, 2) a new disk was added, or 3) the write
> > operation
> > > > > > was only partially successful. How do you plan to handle this
> case?
> > > > > > Consider that the controller and the broker can run on the same
> JVM
> > > > > > and use a log directory different from the metadata log
> directory.
> > > > > > Controllers only read and write to the metadata log directory.
> > > > > >
> > > > > > JS8: In the KIP you mention "Learning of a HWM from the leader,
> > which
> > > > > > the leader allows for because it will send valid fetch responses
> > back
> > > > > > to nodes that do not have a cluster id." One implementation
> > > complexity
> > > > > > is that KRaft can discover the HWM and send a handleCommit event
> > > > > > without having fetched all data up to the HWM. What KRaft
> > guarantees
> > > > > > is that the active leader will not receive a handleLeaderChange
> > event
> > > > > > until it has caught up to the leader's epoch. How do you plan to
> > > > > > implement this?
> > > > > >
> > > > > > Thanks,
> > > > > > --
> > > > > > -José
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi Jun,

RE JR12:
Dynamic quorums can technically not require formatting, but I fear it will
cause cluster misconfigurations more easily than the static quorum case.
The main difference between the two quorum deployments is that static
quorum cannot recover from data loss on a voter, whereas dynamic quorum
can. Below is my reasoning for why maintaining this requirement for dynamic
clusters is better for Kafka operators given that formatting each node's
disk is currently required.

The KRaft voter set in both static and dynamic deployments is a piece of
bootstrapping data that requires some orchestration to manage. Currently,
the static voter set is managed by the supplier of a `.properties` file
which contains `controller.quorum.voters`. In the static quorum case, the
requirements from kafka on an orchestration layer to safely manage the
value of `controller.quorum.voters` config is pretty straightforward:
supply the same value on all nodes all the time forever. I think it is more
obvious to operators, without needing to know too much about how KRaft
works, that having different values for `controller.quorum.voters` is
incorrect and unsafe.

The dynamic voter set's contents is initially managed by the caller of
`kafka-storage format`, and then it is managed by KRaft itself. I will just
focus on what a standalone dynamic controller deployment would look like
without formatting, but bootstrapping a dynamic quorum with multiple
controllers is unsafe for the same reasons. In order to remove the
formatting requirement for dynamic clusters, we could imagine having
something like `controller.quorum.standalone.enabled`. When that config is
defined during startup, kafka writes the bootstrapping VotersRecord and
KRaftVersion that would be done during formatting. However, the
requirements from kafka on an orchestration layer to safely manage this
"standalone" config are more complicated than the static quorum case. They
are also not obvious without prior knowledge of bootstrapping quorums. This
config should only be set on one node whenever the cluster has not been
"bootstrapped," and it cannot be present on any nodes during startup if the
cluster has already been "bootstrapped." If this config is present after
the cluster has a voter set, it can result in multiple KRaft leaders if a
node with the standalone config defined experiences data loss and tries to
restart.

This "unsafeness" WRT dynamic quorum also applies to `kafka-storage
format,` but that CLI does not bootstrap with dynamic quorum by default, as
the user has to specify one of `--standalone`, `--initial-controllers`, or
`--no-initial-controllers` when `controller.quorum.voters` is not defined.
Maybe that is a sufficient argument that it is okay to introduce the
`controller.quorum.standalone.enabled` static config as part of this KIP.
Interested to know what you think about this.

Best,
Kevin Wu

On Tue, Mar 31, 2026 at 11:34 AM Jun Rao via dev <de...@kafka.apache.org>
wrote:

> Hi, Kevin,
>
> Thanks for the reply.
>
> JR12. https://kafka.apache.org/42/operations/kraft/ specifies two
> deployment methods for KRaft: static and dynamic. Are you saying that
> dynamic still requires formatting while static doesn't? Could you explain
> why there is difference? BTW, which method do we recommend?
>
> Jun
>
> On Mon, Mar 30, 2026 at 9:03 AM Kevin Wu <ke...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the reply.
> >
> > RE JR10: Yes, I will update the KIP to reflect that.
> >
> > RE JR11: Yeah, I think it is fine to write V2.
> >
> > RE JR 12: Is this in reference to KIP-853: Dynamic Quorum
> Reconfiguration?
> > If so, see: "However, operators still have the option to format nodes to
> > set the MV, feature versions, scram credentials, or to properly
> provision a
> > kraft.version=1 cluster." In order to bootstrap any dynamic quorum (i.e.
> > kraft.version=1) with an initial voter set, it is required to format a
> > controller(s) with either `--standalone` or `--initial-controllers` so
> that
> > a KRaft VotersRecord is part of the 0-0.checkpoint. Formatting
> controllers
> > is still needed if you want to specify a non-default feature level or
> > metadata version, and kraft.version=1 would be a "non-default" KRaft
> > version (mainly because it is not correct without formatting, described
> > below).
> >
> > I'm not sure if removing this formatting requirement for new KIP-853
> > clusters is in-scope for this KIP. The main issue with this is: How does
> a
> > node know it can safely write a "bootstrapping" 0-0.checkpoint with the
> > KRaft VotersRecord on startup of the kafka process without knowing any
> > state of the cluster? This can lead to split-brain when a node writes
> this
> > for a cluster who has already elected a leader. Currently, the caller of
> > the kafka-storage format command is responsible for writing this exactly
> > once for the lifetime of the cluster.
> >
> > Operators still have the option of starting kafka without formatting, and
> > then upgrading the kraft version to kraft.version=1. This path allows
> for a
> > dynamic quorum without formatting the cluster.
> >
> > Thanks,
> > Kevin Wu
> >
> > On Fri, Mar 27, 2026 at 4:20 PM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> > > Hi, Kevin,
> > >
> > > Thanks for the updated KIP. It's better if we can remove the formatting
> > > requirements for all nodes.
> > >
> > > JR10. "The reason for this KIP is to remove the requirement of brokers
> > > needing to run kafka-storage format  before starting Kafka."
> > > Should we change brokers to nodes?
> > >
> > > JR11. "When --cluster-id  is specified, the formatter writes
> > > meta.properties  V1."
> > > It's a bit weird for the new code to write in V1 format. Could it write
> > in
> > > V2 format?
> > >
> > > JR12. Without formatting, is it true that one can only bootstrap a
> > > standalone controller? In other words, does bootstrapping with multiple
> > > controllers still require formatting?
> > >
> > > Jun
> > >
> > > On Thu, Mar 19, 2026 at 1:39 AM Kevin Wu <ke...@gmail.com>
> wrote:
> > >
> > > > Hi José,
> > > >
> > > > Thanks for the replies and questions.
> > > >
> > > > RE JS1: "Can you clarify that this KIP removes the need for all Kafka
> > > nodes
> > > > to be formatted prior to starting Kafka." Hmmm, I guess in the static
> > > > cluster case that skips formatting having a newer software version +
> > > older
> > > > MV is not a possible case, so I will remove that mention from the
> KIP.
> > We
> > > > should default to the latest MV if we skip formatting, which will
> > support
> > > > writing a ClusterIdRecord.
> > > >
> > > > Right now, it is not completely clear to me how we can allow
> bootstrap
> > > > controllers (this applies mainly for kraft.version=0, since
> > > kraft.version=1
> > > > cannot elect a leader without proper formatting) to also skip
> > formatting.
> > > > That is why I said in the proposed changes: "*Remove the requirement
> of
> > > > brokers and observer controllers to format before starting kafka"*. I
> > > agree
> > > > that KRaft can still elect a leader without clusterId in this case,
> but
> > > I'm
> > > > not completely sure how a QuorumController with an "empty" clusterId
> > > which
> > > > needs to be set later, should behave. My working idea is detailed in
> RE
> > > > JS6. This is required because the active controller needs to
> generate a
> > > > clusterId and write it back to KRaft upon activation in order for the
> > > > committed `ClusterIdRecord` to appear in records passed to
> > > > `RaftListener#handleCommit()`, so we cannot block its initialization.
> > > > Keeping the assumption that QuorumController.clusterId is final and
> > > > non-null would be nice, but that requires all KRaft voters to format
> > > with a
> > > > cluster.id. Let me know what you think about the best way to remove
> > this
> > > > requirement.
> > > >
> > > > RE JS2: My plan was to continue to write meta.properties V1 during
> > > > formatting with a `cluster.id` field like today, but also write a
> > > > `ClusterIdRecord` to the bootstrap snapshot for redundancy if the MV
> > > > supports it (I'm not sure if kafka is expected to handle only partial
> > log
> > > > directory corruption/destruction). If the "bootstrap controller
> cluster
> > > id
> > > > check" from JS4 is correct, then the initial active controller is
> > > > guaranteed to have a non-null `cluster.id` in meta.properties. So
> long
> > > as
> > > > the MV supports it, the active controller would then write
> > > ClusterIdRecord
> > > > as part of the bootstrap records.
> > > >
> > > > RE JS3: When I said this, I meant that the restriction of waiting for
> > the
> > > > discovery of cluster.id to persist it to meta.properties during
> broker
> > > > startup is no more restrictive than what already currently exists,
> > which
> > > is
> > > > being caught up to the HWM in order to register with the active
> > > controller.
> > > >
> > > > RE JS 4: Yeah, I thought about this, specifically around the
> > > > kraft.version=1 case since it is less straightforward what a
> "bootstrap
> > > > controller" is. Under the current design, in kraft.version=0, any
> node
> > > who
> > > > is part of the `controller.quorum.voters` config must have
> > > > `meta.properties` with `cluster.id`. In kraft.version=1, any node
> who
> > > has
> > > > a
> > > > `0-0.checkpoint` is considered a "bootstrap controller." This is a
> > > > heuristic, but I believe it is correct, since in order for the
> > > > 0-0.checkpoint to not exist on a node which formatted with
> --standalone
> > > or
> > > > --initial-controllers, there must have either been another checkpoint
> > > with
> > > > committed records, which imply an elected initial leader, or a disk
> > loss.
> > > > Whenever a voter with id X and initial directory-id A comes back as
> (X,
> > > B),
> > > > this process incarnation is an observer from the perspective of
> KRaft,
> > > and
> > > > I think we can assume it has neither `meta.properties` or
> > > `0-0.checkpoint`
> > > > if the operator did not format it (assumption from RE JS2 about the
> > kinds
> > > > of storage failures we expect to handle are not partial directory
> > > > failures). In this case, the "bootstrap controller" check does not
> > apply
> > > to
> > > > (X, B), and if auto-join is enabled, it will follow the steps
> detailed
> > in
> > > > RE JS5 to recover and rejoin the voter set. If we remove the
> > requirement
> > > on
> > > > all nodes to format, then we would not need to implement these
> checks.
> > > >
> > > > RE JS5: An observer without clusterId who can auto-join will fetch
> > until
> > > > its KafkaRaftClient updates the cluster id in-memory (basically,
> > > auto-join
> > > > is off until it discovers the leader's clusterId). If the observer
> has
> > > > clusterId, it needs to match the leader's to perform a successful
> > fetch,
> > > > which is required for successfully adding a voter via auto-join.
> > > >
> > > > RE JS6: Apologies, I meant to say a MetadataPublisher registered to
> the
> > > > MetadataLoader. Although, looking at this again, maybe this
> discovery +
> > > > persistence of clusterId can be handled by a new RaftListener
> instead.
> > I
> > > > don't think we need the overhead of the MetadataImage + MetadataDelta
> > for
> > > > this feature since a RaftListener's `handleCommit()` and
> > > > `handleLoadSnapshot()` contain `ClusterIdRecord`. However, this means
> > > > needing a third listener besides the MetadataLoader and
> > > QuorumMetaListener,
> > > > and therefore an additional call to log#read() when handling KRaft
> > > commits
> > > > + snapshots. From my reading, it seems like the Kafka log layer does
> > not
> > > > attempt any caching, and instead we rely on the OS page cache.
> Because
> > of
> > > > this, I think we should be using MetadataPublisher, but let me know
> > what
> > > > you think.
> > > >
> > > > I am thinking of using an AtomicReference<String> to represent the
> > > > clusterId in-memory. This RaftListener/MetadataPublisher will be the
> > only
> > > > writer to this value if it is not already defined by meta.properties,
> > but
> > > > there are many readers of this value. The initial value of this
> > reference
> > > > is null or the cluster.id from meta.properties. Upon reading
> > > > `ClusterIdRecord`, the listener will throw an exception if it has a
> > > > non-null clusterId and reads a ClusterIdRecord with a different ID.
> If
> > it
> > > > does not have cluster.id set and reads a ClusterIdRecord, it will
> > update
> > > > the AtomicReference and persist cluster.id to meta.properties. Let
> me
> > > know
> > > > if this approach sounds reasonable to you.
> > > >
> > > > RE JS7: From what I understand about MetaPropertiesEnsemble and its
> > > > verify() method, I think it is reasonable to say our
> > > > RaftListener/MetadataPublisher will know how many (if any)
> > > > `meta.properties` files it is responsible for persisting cluster.id
> to
> > > > during the current process incarnation when it starts up. Currently
> we
> > > only
> > > > validate the MetaPropertiesEnsemble in two places: during formatting,
> > and
> > > > during node startup. From what I understand, scenarios 1 and 2 should
> > > only
> > > > occur alongside a restart of the kafka process (to generate a new
> > > > directory-id and/or update log.dirs), but please correct me if this
> > > > assumption is wrong. I'm not sure if scenario 3 is referring to a
> > partial
> > > > write of a given meta.properties (i.e. it does not contain
> cluster.id
> > ),
> > > or
> > > > not writing the discovered cluster.id to all meta.properties files
> on
> > > the
> > > > node before a crash. If a meta.properties does not exist in a
> > > log/metadata
> > > > log directory during startup, we need to write a V2 one without a
> > > > cluster.id,
> > > > but we would be aware of this. If we succeed writing cluster.id to
> at
> > > > least
> > > > one meta.properties via the ClusterIdRecord, I believe it is safe to
> > > write
> > > > that same value to the other meta.properties upon restart if they
> exist
> > > > because cluster.id does not change.
> > > >
> > > > I may have previously removed this from the KIP, but given this
> > > discussion,
> > > > I believe it is only safe to update the in-memory cluster.id only
> > after
> > > > writing this to all meta.properties on a node.
> > > >
> > > > RE JS8: Okay, maybe I will just rewrite the section. My point was to
> > say
> > > > something like: a node's discovery of the leader's committed
> > cluster.id
> > > > relies on the discovery of a HWM and our
> RaftListener/MetadataPublisher
> > > to
> > > > be registered with the raft client, and that we need to wait for
> these
> > > > things before the startup logic in Controller/BrokerServer executes.
> > > > However, if our listener does not see the ClusterIdRecord in
> > > `handleCommit`
> > > > or `handleLoadSnapshot`, it can't do anything meaningful, so it is
> more
> > > > accurate to say we need to wait until ClusterIdRecord is committed.
> > > >
> > > > On Thu, Mar 19, 2026 at 12:57 AM José Armando García Sancio via dev <
> > > > dev@kafka.apache.org> wrote:
> > > >
> > > > > Hi Kevin, Thanks for the KIP and excuse my delay response.
> > > > >
> > > > > JS1: Can you clarify that this KIP removes the need for all Kafka
> > > > > nodes to be formatted pior to starting Kafka. However, this doesn't
> > > > > prevent users from formatting their broker with a cluster ID if
> they
> > > > > prefer. This is especially needed for Kafka nodes formatted for a
> > > > > cluster using an MV that doesn't support this feature.
> > > > >
> > > > > JS2: How are you planning to implement "kafka-storage format
> > > > > --clusterid YYY --standalone"? Is that going to behave like it does
> > > > > today by writing the cluster id to the meta.properties files? Or
> are
> > > > > you planning to write the cluster id using the ClusterIdRecord to
> the
> > > > > bootstrap.checkpoint or 0-0.checkpoint (after KIP-1170)?
> > > > >
> > > > > JS3: In one of your replies you say "Discovering the cluster id
> value
> > > > > for the first time would only require a single FetchSnapshot or a
> > > > > Fetch of the bootstrap metadata records." This is not entirely
> > > > > accurate. The best we can say is that brokers need to catch up to
> the
> > > > > HWM before they can send a registration requests to the active
> > > > > controller or it can start a few internal component. However, the
> > > > > broker already had this requirement prior to this KIP, so it is not
> > > > > new.
> > > > >
> > > > > JS4: In the KIP you mention "if meta.properties does not exist and
> > the
> > > > > node is a bootstrap controller, throw a runtime exception." Can you
> > > > > explain how you plan to implement this? One important aspect to
> > > > > consider is that in KRaft voters (controllers) are identified by
> the
> > > > > node ID and directory ID. A node can recover from a disk failure by
> > > > > coming back with the same node ID but a different directory ID. In
> > > > > this case, the controller should auto-recover if the auto-join
> > feature
> > > > > is enabled.
> > > > >
> > > > > JS5: In the KIP you mention "One detail here is that observer
> > > > > controllers with auto-join must wait until they have a cluster id
> > > > > before trying to add or remove themselves." I understand the reason
> > > > > for this requirement. If a node auto-joins the controller cluster,
> > you
> > > > > must guarantee that it knows the cluster id in case it becomes the
> > > > > leader and needs to write the ClusterIDRecord. Can you elaborate on
> > > > > your implementation plan?
> > > > >
> > > > > JS6: In the KIP you mention "This can be implemented as a
> > > > > MetadataPublisher that registers to the raft client alongside the
> > > > > MetadataLoader." Metadata publishers don't register with the KRaft
> > > > > client. RaftClient.Listener register with the KRaft client.
> Metadata
> > > > > publisher register with the metadata loader instead.
> > > > >
> > > > > JS7: One complexity is that there is a meta.properties per log
> > > > > directory and metadata log directory. This means that in the stable
> > > > > case the cluster ID exists in all the meta.properties files.
> > > > > Unfortunately, this may not be the case for several reasons: 1) the
> > > > > disk was replaced, 2) a new disk was added, or 3) the write
> operation
> > > > > was only partially successful. How do you plan to handle this case?
> > > > > Consider that the controller and the broker can run on the same JVM
> > > > > and use a log directory different from the metadata log directory.
> > > > > Controllers only read and write to the metadata log directory.
> > > > >
> > > > > JS8: In the KIP you mention "Learning of a HWM from the leader,
> which
> > > > > the leader allows for because it will send valid fetch responses
> back
> > > > > to nodes that do not have a cluster id." One implementation
> > complexity
> > > > > is that KRaft can discover the HWM and send a handleCommit event
> > > > > without having fetched all data up to the HWM. What KRaft
> guarantees
> > > > > is that the active leader will not receive a handleLeaderChange
> event
> > > > > until it has caught up to the leader's epoch. How do you plan to
> > > > > implement this?
> > > > >
> > > > > Thanks,
> > > > > --
> > > > > -José
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Jun Rao via dev <de...@kafka.apache.org>]

Hi, Kevin,

Thanks for the reply.

JR12. https://kafka.apache.org/42/operations/kraft/ specifies two
deployment methods for KRaft: static and dynamic. Are you saying that
dynamic still requires formatting while static doesn't? Could you explain
why there is difference? BTW, which method do we recommend?

Jun

On Mon, Mar 30, 2026 at 9:03 AM Kevin Wu <ke...@gmail.com> wrote:

> Hi Jun,
>
> Thanks for the reply.
>
> RE JR10: Yes, I will update the KIP to reflect that.
>
> RE JR11: Yeah, I think it is fine to write V2.
>
> RE JR 12: Is this in reference to KIP-853: Dynamic Quorum Reconfiguration?
> If so, see: "However, operators still have the option to format nodes to
> set the MV, feature versions, scram credentials, or to properly provision a
> kraft.version=1 cluster." In order to bootstrap any dynamic quorum (i.e.
> kraft.version=1) with an initial voter set, it is required to format a
> controller(s) with either `--standalone` or `--initial-controllers` so that
> a KRaft VotersRecord is part of the 0-0.checkpoint. Formatting controllers
> is still needed if you want to specify a non-default feature level or
> metadata version, and kraft.version=1 would be a "non-default" KRaft
> version (mainly because it is not correct without formatting, described
> below).
>
> I'm not sure if removing this formatting requirement for new KIP-853
> clusters is in-scope for this KIP. The main issue with this is: How does a
> node know it can safely write a "bootstrapping" 0-0.checkpoint with the
> KRaft VotersRecord on startup of the kafka process without knowing any
> state of the cluster? This can lead to split-brain when a node writes this
> for a cluster who has already elected a leader. Currently, the caller of
> the kafka-storage format command is responsible for writing this exactly
> once for the lifetime of the cluster.
>
> Operators still have the option of starting kafka without formatting, and
> then upgrading the kraft version to kraft.version=1. This path allows for a
> dynamic quorum without formatting the cluster.
>
> Thanks,
> Kevin Wu
>
> On Fri, Mar 27, 2026 at 4:20 PM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi, Kevin,
> >
> > Thanks for the updated KIP. It's better if we can remove the formatting
> > requirements for all nodes.
> >
> > JR10. "The reason for this KIP is to remove the requirement of brokers
> > needing to run kafka-storage format  before starting Kafka."
> > Should we change brokers to nodes?
> >
> > JR11. "When --cluster-id  is specified, the formatter writes
> > meta.properties  V1."
> > It's a bit weird for the new code to write in V1 format. Could it write
> in
> > V2 format?
> >
> > JR12. Without formatting, is it true that one can only bootstrap a
> > standalone controller? In other words, does bootstrapping with multiple
> > controllers still require formatting?
> >
> > Jun
> >
> > On Thu, Mar 19, 2026 at 1:39 AM Kevin Wu <ke...@gmail.com> wrote:
> >
> > > Hi José,
> > >
> > > Thanks for the replies and questions.
> > >
> > > RE JS1: "Can you clarify that this KIP removes the need for all Kafka
> > nodes
> > > to be formatted prior to starting Kafka." Hmmm, I guess in the static
> > > cluster case that skips formatting having a newer software version +
> > older
> > > MV is not a possible case, so I will remove that mention from the KIP.
> We
> > > should default to the latest MV if we skip formatting, which will
> support
> > > writing a ClusterIdRecord.
> > >
> > > Right now, it is not completely clear to me how we can allow bootstrap
> > > controllers (this applies mainly for kraft.version=0, since
> > kraft.version=1
> > > cannot elect a leader without proper formatting) to also skip
> formatting.
> > > That is why I said in the proposed changes: "*Remove the requirement of
> > > brokers and observer controllers to format before starting kafka"*. I
> > agree
> > > that KRaft can still elect a leader without clusterId in this case, but
> > I'm
> > > not completely sure how a QuorumController with an "empty" clusterId
> > which
> > > needs to be set later, should behave. My working idea is detailed in RE
> > > JS6. This is required because the active controller needs to generate a
> > > clusterId and write it back to KRaft upon activation in order for the
> > > committed `ClusterIdRecord` to appear in records passed to
> > > `RaftListener#handleCommit()`, so we cannot block its initialization.
> > > Keeping the assumption that QuorumController.clusterId is final and
> > > non-null would be nice, but that requires all KRaft voters to format
> > with a
> > > cluster.id. Let me know what you think about the best way to remove
> this
> > > requirement.
> > >
> > > RE JS2: My plan was to continue to write meta.properties V1 during
> > > formatting with a `cluster.id` field like today, but also write a
> > > `ClusterIdRecord` to the bootstrap snapshot for redundancy if the MV
> > > supports it (I'm not sure if kafka is expected to handle only partial
> log
> > > directory corruption/destruction). If the "bootstrap controller cluster
> > id
> > > check" from JS4 is correct, then the initial active controller is
> > > guaranteed to have a non-null `cluster.id` in meta.properties. So long
> > as
> > > the MV supports it, the active controller would then write
> > ClusterIdRecord
> > > as part of the bootstrap records.
> > >
> > > RE JS3: When I said this, I meant that the restriction of waiting for
> the
> > > discovery of cluster.id to persist it to meta.properties during broker
> > > startup is no more restrictive than what already currently exists,
> which
> > is
> > > being caught up to the HWM in order to register with the active
> > controller.
> > >
> > > RE JS 4: Yeah, I thought about this, specifically around the
> > > kraft.version=1 case since it is less straightforward what a "bootstrap
> > > controller" is. Under the current design, in kraft.version=0, any node
> > who
> > > is part of the `controller.quorum.voters` config must have
> > > `meta.properties` with `cluster.id`. In kraft.version=1, any node who
> > has
> > > a
> > > `0-0.checkpoint` is considered a "bootstrap controller." This is a
> > > heuristic, but I believe it is correct, since in order for the
> > > 0-0.checkpoint to not exist on a node which formatted with --standalone
> > or
> > > --initial-controllers, there must have either been another checkpoint
> > with
> > > committed records, which imply an elected initial leader, or a disk
> loss.
> > > Whenever a voter with id X and initial directory-id A comes back as (X,
> > B),
> > > this process incarnation is an observer from the perspective of KRaft,
> > and
> > > I think we can assume it has neither `meta.properties` or
> > `0-0.checkpoint`
> > > if the operator did not format it (assumption from RE JS2 about the
> kinds
> > > of storage failures we expect to handle are not partial directory
> > > failures). In this case, the "bootstrap controller" check does not
> apply
> > to
> > > (X, B), and if auto-join is enabled, it will follow the steps detailed
> in
> > > RE JS5 to recover and rejoin the voter set. If we remove the
> requirement
> > on
> > > all nodes to format, then we would not need to implement these checks.
> > >
> > > RE JS5: An observer without clusterId who can auto-join will fetch
> until
> > > its KafkaRaftClient updates the cluster id in-memory (basically,
> > auto-join
> > > is off until it discovers the leader's clusterId). If the observer has
> > > clusterId, it needs to match the leader's to perform a successful
> fetch,
> > > which is required for successfully adding a voter via auto-join.
> > >
> > > RE JS6: Apologies, I meant to say a MetadataPublisher registered to the
> > > MetadataLoader. Although, looking at this again, maybe this discovery +
> > > persistence of clusterId can be handled by a new RaftListener instead.
> I
> > > don't think we need the overhead of the MetadataImage + MetadataDelta
> for
> > > this feature since a RaftListener's `handleCommit()` and
> > > `handleLoadSnapshot()` contain `ClusterIdRecord`. However, this means
> > > needing a third listener besides the MetadataLoader and
> > QuorumMetaListener,
> > > and therefore an additional call to log#read() when handling KRaft
> > commits
> > > + snapshots. From my reading, it seems like the Kafka log layer does
> not
> > > attempt any caching, and instead we rely on the OS page cache. Because
> of
> > > this, I think we should be using MetadataPublisher, but let me know
> what
> > > you think.
> > >
> > > I am thinking of using an AtomicReference<String> to represent the
> > > clusterId in-memory. This RaftListener/MetadataPublisher will be the
> only
> > > writer to this value if it is not already defined by meta.properties,
> but
> > > there are many readers of this value. The initial value of this
> reference
> > > is null or the cluster.id from meta.properties. Upon reading
> > > `ClusterIdRecord`, the listener will throw an exception if it has a
> > > non-null clusterId and reads a ClusterIdRecord with a different ID. If
> it
> > > does not have cluster.id set and reads a ClusterIdRecord, it will
> update
> > > the AtomicReference and persist cluster.id to meta.properties. Let me
> > know
> > > if this approach sounds reasonable to you.
> > >
> > > RE JS7: From what I understand about MetaPropertiesEnsemble and its
> > > verify() method, I think it is reasonable to say our
> > > RaftListener/MetadataPublisher will know how many (if any)
> > > `meta.properties` files it is responsible for persisting cluster.id to
> > > during the current process incarnation when it starts up. Currently we
> > only
> > > validate the MetaPropertiesEnsemble in two places: during formatting,
> and
> > > during node startup. From what I understand, scenarios 1 and 2 should
> > only
> > > occur alongside a restart of the kafka process (to generate a new
> > > directory-id and/or update log.dirs), but please correct me if this
> > > assumption is wrong. I'm not sure if scenario 3 is referring to a
> partial
> > > write of a given meta.properties (i.e. it does not contain cluster.id
> ),
> > or
> > > not writing the discovered cluster.id to all meta.properties files on
> > the
> > > node before a crash. If a meta.properties does not exist in a
> > log/metadata
> > > log directory during startup, we need to write a V2 one without a
> > > cluster.id,
> > > but we would be aware of this. If we succeed writing cluster.id to at
> > > least
> > > one meta.properties via the ClusterIdRecord, I believe it is safe to
> > write
> > > that same value to the other meta.properties upon restart if they exist
> > > because cluster.id does not change.
> > >
> > > I may have previously removed this from the KIP, but given this
> > discussion,
> > > I believe it is only safe to update the in-memory cluster.id only
> after
> > > writing this to all meta.properties on a node.
> > >
> > > RE JS8: Okay, maybe I will just rewrite the section. My point was to
> say
> > > something like: a node's discovery of the leader's committed
> cluster.id
> > > relies on the discovery of a HWM and our RaftListener/MetadataPublisher
> > to
> > > be registered with the raft client, and that we need to wait for these
> > > things before the startup logic in Controller/BrokerServer executes.
> > > However, if our listener does not see the ClusterIdRecord in
> > `handleCommit`
> > > or `handleLoadSnapshot`, it can't do anything meaningful, so it is more
> > > accurate to say we need to wait until ClusterIdRecord is committed.
> > >
> > > On Thu, Mar 19, 2026 at 12:57 AM José Armando García Sancio via dev <
> > > dev@kafka.apache.org> wrote:
> > >
> > > > Hi Kevin, Thanks for the KIP and excuse my delay response.
> > > >
> > > > JS1: Can you clarify that this KIP removes the need for all Kafka
> > > > nodes to be formatted pior to starting Kafka. However, this doesn't
> > > > prevent users from formatting their broker with a cluster ID if they
> > > > prefer. This is especially needed for Kafka nodes formatted for a
> > > > cluster using an MV that doesn't support this feature.
> > > >
> > > > JS2: How are you planning to implement "kafka-storage format
> > > > --clusterid YYY --standalone"? Is that going to behave like it does
> > > > today by writing the cluster id to the meta.properties files? Or are
> > > > you planning to write the cluster id using the ClusterIdRecord to the
> > > > bootstrap.checkpoint or 0-0.checkpoint (after KIP-1170)?
> > > >
> > > > JS3: In one of your replies you say "Discovering the cluster id value
> > > > for the first time would only require a single FetchSnapshot or a
> > > > Fetch of the bootstrap metadata records." This is not entirely
> > > > accurate. The best we can say is that brokers need to catch up to the
> > > > HWM before they can send a registration requests to the active
> > > > controller or it can start a few internal component. However, the
> > > > broker already had this requirement prior to this KIP, so it is not
> > > > new.
> > > >
> > > > JS4: In the KIP you mention "if meta.properties does not exist and
> the
> > > > node is a bootstrap controller, throw a runtime exception." Can you
> > > > explain how you plan to implement this? One important aspect to
> > > > consider is that in KRaft voters (controllers) are identified by the
> > > > node ID and directory ID. A node can recover from a disk failure by
> > > > coming back with the same node ID but a different directory ID. In
> > > > this case, the controller should auto-recover if the auto-join
> feature
> > > > is enabled.
> > > >
> > > > JS5: In the KIP you mention "One detail here is that observer
> > > > controllers with auto-join must wait until they have a cluster id
> > > > before trying to add or remove themselves." I understand the reason
> > > > for this requirement. If a node auto-joins the controller cluster,
> you
> > > > must guarantee that it knows the cluster id in case it becomes the
> > > > leader and needs to write the ClusterIDRecord. Can you elaborate on
> > > > your implementation plan?
> > > >
> > > > JS6: In the KIP you mention "This can be implemented as a
> > > > MetadataPublisher that registers to the raft client alongside the
> > > > MetadataLoader." Metadata publishers don't register with the KRaft
> > > > client. RaftClient.Listener register with the KRaft client. Metadata
> > > > publisher register with the metadata loader instead.
> > > >
> > > > JS7: One complexity is that there is a meta.properties per log
> > > > directory and metadata log directory. This means that in the stable
> > > > case the cluster ID exists in all the meta.properties files.
> > > > Unfortunately, this may not be the case for several reasons: 1) the
> > > > disk was replaced, 2) a new disk was added, or 3) the write operation
> > > > was only partially successful. How do you plan to handle this case?
> > > > Consider that the controller and the broker can run on the same JVM
> > > > and use a log directory different from the metadata log directory.
> > > > Controllers only read and write to the metadata log directory.
> > > >
> > > > JS8: In the KIP you mention "Learning of a HWM from the leader, which
> > > > the leader allows for because it will send valid fetch responses back
> > > > to nodes that do not have a cluster id." One implementation
> complexity
> > > > is that KRaft can discover the HWM and send a handleCommit event
> > > > without having fetched all data up to the HWM. What KRaft guarantees
> > > > is that the active leader will not receive a handleLeaderChange event
> > > > until it has caught up to the leader's epoch. How do you plan to
> > > > implement this?
> > > >
> > > > Thanks,
> > > > --
> > > > -José
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi Jun,

Thanks for the reply.

RE JR10: Yes, I will update the KIP to reflect that.

RE JR11: Yeah, I think it is fine to write V2.

RE JR 12: Is this in reference to KIP-853: Dynamic Quorum Reconfiguration?
If so, see: "However, operators still have the option to format nodes to
set the MV, feature versions, scram credentials, or to properly provision a
kraft.version=1 cluster." In order to bootstrap any dynamic quorum (i.e.
kraft.version=1) with an initial voter set, it is required to format a
controller(s) with either `--standalone` or `--initial-controllers` so that
a KRaft VotersRecord is part of the 0-0.checkpoint. Formatting controllers
is still needed if you want to specify a non-default feature level or
metadata version, and kraft.version=1 would be a "non-default" KRaft
version (mainly because it is not correct without formatting, described
below).

I'm not sure if removing this formatting requirement for new KIP-853
clusters is in-scope for this KIP. The main issue with this is: How does a
node know it can safely write a "bootstrapping" 0-0.checkpoint with the
KRaft VotersRecord on startup of the kafka process without knowing any
state of the cluster? This can lead to split-brain when a node writes this
for a cluster who has already elected a leader. Currently, the caller of
the kafka-storage format command is responsible for writing this exactly
once for the lifetime of the cluster.

Operators still have the option of starting kafka without formatting, and
then upgrading the kraft version to kraft.version=1. This path allows for a
dynamic quorum without formatting the cluster.

Thanks,
Kevin Wu

On Fri, Mar 27, 2026 at 4:20 PM Jun Rao via dev <de...@kafka.apache.org>
wrote:

> Hi, Kevin,
>
> Thanks for the updated KIP. It's better if we can remove the formatting
> requirements for all nodes.
>
> JR10. "The reason for this KIP is to remove the requirement of brokers
> needing to run kafka-storage format  before starting Kafka."
> Should we change brokers to nodes?
>
> JR11. "When --cluster-id  is specified, the formatter writes
> meta.properties  V1."
> It's a bit weird for the new code to write in V1 format. Could it write in
> V2 format?
>
> JR12. Without formatting, is it true that one can only bootstrap a
> standalone controller? In other words, does bootstrapping with multiple
> controllers still require formatting?
>
> Jun
>
> On Thu, Mar 19, 2026 at 1:39 AM Kevin Wu <ke...@gmail.com> wrote:
>
> > Hi José,
> >
> > Thanks for the replies and questions.
> >
> > RE JS1: "Can you clarify that this KIP removes the need for all Kafka
> nodes
> > to be formatted prior to starting Kafka." Hmmm, I guess in the static
> > cluster case that skips formatting having a newer software version +
> older
> > MV is not a possible case, so I will remove that mention from the KIP. We
> > should default to the latest MV if we skip formatting, which will support
> > writing a ClusterIdRecord.
> >
> > Right now, it is not completely clear to me how we can allow bootstrap
> > controllers (this applies mainly for kraft.version=0, since
> kraft.version=1
> > cannot elect a leader without proper formatting) to also skip formatting.
> > That is why I said in the proposed changes: "*Remove the requirement of
> > brokers and observer controllers to format before starting kafka"*. I
> agree
> > that KRaft can still elect a leader without clusterId in this case, but
> I'm
> > not completely sure how a QuorumController with an "empty" clusterId
> which
> > needs to be set later, should behave. My working idea is detailed in RE
> > JS6. This is required because the active controller needs to generate a
> > clusterId and write it back to KRaft upon activation in order for the
> > committed `ClusterIdRecord` to appear in records passed to
> > `RaftListener#handleCommit()`, so we cannot block its initialization.
> > Keeping the assumption that QuorumController.clusterId is final and
> > non-null would be nice, but that requires all KRaft voters to format
> with a
> > cluster.id. Let me know what you think about the best way to remove this
> > requirement.
> >
> > RE JS2: My plan was to continue to write meta.properties V1 during
> > formatting with a `cluster.id` field like today, but also write a
> > `ClusterIdRecord` to the bootstrap snapshot for redundancy if the MV
> > supports it (I'm not sure if kafka is expected to handle only partial log
> > directory corruption/destruction). If the "bootstrap controller cluster
> id
> > check" from JS4 is correct, then the initial active controller is
> > guaranteed to have a non-null `cluster.id` in meta.properties. So long
> as
> > the MV supports it, the active controller would then write
> ClusterIdRecord
> > as part of the bootstrap records.
> >
> > RE JS3: When I said this, I meant that the restriction of waiting for the
> > discovery of cluster.id to persist it to meta.properties during broker
> > startup is no more restrictive than what already currently exists, which
> is
> > being caught up to the HWM in order to register with the active
> controller.
> >
> > RE JS 4: Yeah, I thought about this, specifically around the
> > kraft.version=1 case since it is less straightforward what a "bootstrap
> > controller" is. Under the current design, in kraft.version=0, any node
> who
> > is part of the `controller.quorum.voters` config must have
> > `meta.properties` with `cluster.id`. In kraft.version=1, any node who
> has
> > a
> > `0-0.checkpoint` is considered a "bootstrap controller." This is a
> > heuristic, but I believe it is correct, since in order for the
> > 0-0.checkpoint to not exist on a node which formatted with --standalone
> or
> > --initial-controllers, there must have either been another checkpoint
> with
> > committed records, which imply an elected initial leader, or a disk loss.
> > Whenever a voter with id X and initial directory-id A comes back as (X,
> B),
> > this process incarnation is an observer from the perspective of KRaft,
> and
> > I think we can assume it has neither `meta.properties` or
> `0-0.checkpoint`
> > if the operator did not format it (assumption from RE JS2 about the kinds
> > of storage failures we expect to handle are not partial directory
> > failures). In this case, the "bootstrap controller" check does not apply
> to
> > (X, B), and if auto-join is enabled, it will follow the steps detailed in
> > RE JS5 to recover and rejoin the voter set. If we remove the requirement
> on
> > all nodes to format, then we would not need to implement these checks.
> >
> > RE JS5: An observer without clusterId who can auto-join will fetch until
> > its KafkaRaftClient updates the cluster id in-memory (basically,
> auto-join
> > is off until it discovers the leader's clusterId). If the observer has
> > clusterId, it needs to match the leader's to perform a successful fetch,
> > which is required for successfully adding a voter via auto-join.
> >
> > RE JS6: Apologies, I meant to say a MetadataPublisher registered to the
> > MetadataLoader. Although, looking at this again, maybe this discovery +
> > persistence of clusterId can be handled by a new RaftListener instead. I
> > don't think we need the overhead of the MetadataImage + MetadataDelta for
> > this feature since a RaftListener's `handleCommit()` and
> > `handleLoadSnapshot()` contain `ClusterIdRecord`. However, this means
> > needing a third listener besides the MetadataLoader and
> QuorumMetaListener,
> > and therefore an additional call to log#read() when handling KRaft
> commits
> > + snapshots. From my reading, it seems like the Kafka log layer does not
> > attempt any caching, and instead we rely on the OS page cache. Because of
> > this, I think we should be using MetadataPublisher, but let me know what
> > you think.
> >
> > I am thinking of using an AtomicReference<String> to represent the
> > clusterId in-memory. This RaftListener/MetadataPublisher will be the only
> > writer to this value if it is not already defined by meta.properties, but
> > there are many readers of this value. The initial value of this reference
> > is null or the cluster.id from meta.properties. Upon reading
> > `ClusterIdRecord`, the listener will throw an exception if it has a
> > non-null clusterId and reads a ClusterIdRecord with a different ID. If it
> > does not have cluster.id set and reads a ClusterIdRecord, it will update
> > the AtomicReference and persist cluster.id to meta.properties. Let me
> know
> > if this approach sounds reasonable to you.
> >
> > RE JS7: From what I understand about MetaPropertiesEnsemble and its
> > verify() method, I think it is reasonable to say our
> > RaftListener/MetadataPublisher will know how many (if any)
> > `meta.properties` files it is responsible for persisting cluster.id to
> > during the current process incarnation when it starts up. Currently we
> only
> > validate the MetaPropertiesEnsemble in two places: during formatting, and
> > during node startup. From what I understand, scenarios 1 and 2 should
> only
> > occur alongside a restart of the kafka process (to generate a new
> > directory-id and/or update log.dirs), but please correct me if this
> > assumption is wrong. I'm not sure if scenario 3 is referring to a partial
> > write of a given meta.properties (i.e. it does not contain cluster.id),
> or
> > not writing the discovered cluster.id to all meta.properties files on
> the
> > node before a crash. If a meta.properties does not exist in a
> log/metadata
> > log directory during startup, we need to write a V2 one without a
> > cluster.id,
> > but we would be aware of this. If we succeed writing cluster.id to at
> > least
> > one meta.properties via the ClusterIdRecord, I believe it is safe to
> write
> > that same value to the other meta.properties upon restart if they exist
> > because cluster.id does not change.
> >
> > I may have previously removed this from the KIP, but given this
> discussion,
> > I believe it is only safe to update the in-memory cluster.id only after
> > writing this to all meta.properties on a node.
> >
> > RE JS8: Okay, maybe I will just rewrite the section. My point was to say
> > something like: a node's discovery of the leader's committed cluster.id
> > relies on the discovery of a HWM and our RaftListener/MetadataPublisher
> to
> > be registered with the raft client, and that we need to wait for these
> > things before the startup logic in Controller/BrokerServer executes.
> > However, if our listener does not see the ClusterIdRecord in
> `handleCommit`
> > or `handleLoadSnapshot`, it can't do anything meaningful, so it is more
> > accurate to say we need to wait until ClusterIdRecord is committed.
> >
> > On Thu, Mar 19, 2026 at 12:57 AM José Armando García Sancio via dev <
> > dev@kafka.apache.org> wrote:
> >
> > > Hi Kevin, Thanks for the KIP and excuse my delay response.
> > >
> > > JS1: Can you clarify that this KIP removes the need for all Kafka
> > > nodes to be formatted pior to starting Kafka. However, this doesn't
> > > prevent users from formatting their broker with a cluster ID if they
> > > prefer. This is especially needed for Kafka nodes formatted for a
> > > cluster using an MV that doesn't support this feature.
> > >
> > > JS2: How are you planning to implement "kafka-storage format
> > > --clusterid YYY --standalone"? Is that going to behave like it does
> > > today by writing the cluster id to the meta.properties files? Or are
> > > you planning to write the cluster id using the ClusterIdRecord to the
> > > bootstrap.checkpoint or 0-0.checkpoint (after KIP-1170)?
> > >
> > > JS3: In one of your replies you say "Discovering the cluster id value
> > > for the first time would only require a single FetchSnapshot or a
> > > Fetch of the bootstrap metadata records." This is not entirely
> > > accurate. The best we can say is that brokers need to catch up to the
> > > HWM before they can send a registration requests to the active
> > > controller or it can start a few internal component. However, the
> > > broker already had this requirement prior to this KIP, so it is not
> > > new.
> > >
> > > JS4: In the KIP you mention "if meta.properties does not exist and the
> > > node is a bootstrap controller, throw a runtime exception." Can you
> > > explain how you plan to implement this? One important aspect to
> > > consider is that in KRaft voters (controllers) are identified by the
> > > node ID and directory ID. A node can recover from a disk failure by
> > > coming back with the same node ID but a different directory ID. In
> > > this case, the controller should auto-recover if the auto-join feature
> > > is enabled.
> > >
> > > JS5: In the KIP you mention "One detail here is that observer
> > > controllers with auto-join must wait until they have a cluster id
> > > before trying to add or remove themselves." I understand the reason
> > > for this requirement. If a node auto-joins the controller cluster, you
> > > must guarantee that it knows the cluster id in case it becomes the
> > > leader and needs to write the ClusterIDRecord. Can you elaborate on
> > > your implementation plan?
> > >
> > > JS6: In the KIP you mention "This can be implemented as a
> > > MetadataPublisher that registers to the raft client alongside the
> > > MetadataLoader." Metadata publishers don't register with the KRaft
> > > client. RaftClient.Listener register with the KRaft client. Metadata
> > > publisher register with the metadata loader instead.
> > >
> > > JS7: One complexity is that there is a meta.properties per log
> > > directory and metadata log directory. This means that in the stable
> > > case the cluster ID exists in all the meta.properties files.
> > > Unfortunately, this may not be the case for several reasons: 1) the
> > > disk was replaced, 2) a new disk was added, or 3) the write operation
> > > was only partially successful. How do you plan to handle this case?
> > > Consider that the controller and the broker can run on the same JVM
> > > and use a log directory different from the metadata log directory.
> > > Controllers only read and write to the metadata log directory.
> > >
> > > JS8: In the KIP you mention "Learning of a HWM from the leader, which
> > > the leader allows for because it will send valid fetch responses back
> > > to nodes that do not have a cluster id." One implementation complexity
> > > is that KRaft can discover the HWM and send a handleCommit event
> > > without having fetched all data up to the HWM. What KRaft guarantees
> > > is that the active leader will not receive a handleLeaderChange event
> > > until it has caught up to the leader's epoch. How do you plan to
> > > implement this?
> > >
> > > Thanks,
> > > --
> > > -José
> > >
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Jun Rao via dev <de...@kafka.apache.org>]

Hi, Kevin,

Thanks for the updated KIP. It's better if we can remove the formatting
requirements for all nodes.

JR10. "The reason for this KIP is to remove the requirement of brokers
needing to run kafka-storage format  before starting Kafka."
Should we change brokers to nodes?

JR11. "When --cluster-id  is specified, the formatter writes
meta.properties  V1."
It's a bit weird for the new code to write in V1 format. Could it write in
V2 format?

JR12. Without formatting, is it true that one can only bootstrap a
standalone controller? In other words, does bootstrapping with multiple
controllers still require formatting?

Jun

On Thu, Mar 19, 2026 at 1:39 AM Kevin Wu <ke...@gmail.com> wrote:

> Hi José,
>
> Thanks for the replies and questions.
>
> RE JS1: "Can you clarify that this KIP removes the need for all Kafka nodes
> to be formatted prior to starting Kafka." Hmmm, I guess in the static
> cluster case that skips formatting having a newer software version + older
> MV is not a possible case, so I will remove that mention from the KIP. We
> should default to the latest MV if we skip formatting, which will support
> writing a ClusterIdRecord.
>
> Right now, it is not completely clear to me how we can allow bootstrap
> controllers (this applies mainly for kraft.version=0, since kraft.version=1
> cannot elect a leader without proper formatting) to also skip formatting.
> That is why I said in the proposed changes: "*Remove the requirement of
> brokers and observer controllers to format before starting kafka"*. I agree
> that KRaft can still elect a leader without clusterId in this case, but I'm
> not completely sure how a QuorumController with an "empty" clusterId which
> needs to be set later, should behave. My working idea is detailed in RE
> JS6. This is required because the active controller needs to generate a
> clusterId and write it back to KRaft upon activation in order for the
> committed `ClusterIdRecord` to appear in records passed to
> `RaftListener#handleCommit()`, so we cannot block its initialization.
> Keeping the assumption that QuorumController.clusterId is final and
> non-null would be nice, but that requires all KRaft voters to format with a
> cluster.id. Let me know what you think about the best way to remove this
> requirement.
>
> RE JS2: My plan was to continue to write meta.properties V1 during
> formatting with a `cluster.id` field like today, but also write a
> `ClusterIdRecord` to the bootstrap snapshot for redundancy if the MV
> supports it (I'm not sure if kafka is expected to handle only partial log
> directory corruption/destruction). If the "bootstrap controller cluster id
> check" from JS4 is correct, then the initial active controller is
> guaranteed to have a non-null `cluster.id` in meta.properties. So long as
> the MV supports it, the active controller would then write ClusterIdRecord
> as part of the bootstrap records.
>
> RE JS3: When I said this, I meant that the restriction of waiting for the
> discovery of cluster.id to persist it to meta.properties during broker
> startup is no more restrictive than what already currently exists, which is
> being caught up to the HWM in order to register with the active controller.
>
> RE JS 4: Yeah, I thought about this, specifically around the
> kraft.version=1 case since it is less straightforward what a "bootstrap
> controller" is. Under the current design, in kraft.version=0, any node who
> is part of the `controller.quorum.voters` config must have
> `meta.properties` with `cluster.id`. In kraft.version=1, any node who has
> a
> `0-0.checkpoint` is considered a "bootstrap controller." This is a
> heuristic, but I believe it is correct, since in order for the
> 0-0.checkpoint to not exist on a node which formatted with --standalone or
> --initial-controllers, there must have either been another checkpoint with
> committed records, which imply an elected initial leader, or a disk loss.
> Whenever a voter with id X and initial directory-id A comes back as (X, B),
> this process incarnation is an observer from the perspective of KRaft, and
> I think we can assume it has neither `meta.properties` or `0-0.checkpoint`
> if the operator did not format it (assumption from RE JS2 about the kinds
> of storage failures we expect to handle are not partial directory
> failures). In this case, the "bootstrap controller" check does not apply to
> (X, B), and if auto-join is enabled, it will follow the steps detailed in
> RE JS5 to recover and rejoin the voter set. If we remove the requirement on
> all nodes to format, then we would not need to implement these checks.
>
> RE JS5: An observer without clusterId who can auto-join will fetch until
> its KafkaRaftClient updates the cluster id in-memory (basically, auto-join
> is off until it discovers the leader's clusterId). If the observer has
> clusterId, it needs to match the leader's to perform a successful fetch,
> which is required for successfully adding a voter via auto-join.
>
> RE JS6: Apologies, I meant to say a MetadataPublisher registered to the
> MetadataLoader. Although, looking at this again, maybe this discovery +
> persistence of clusterId can be handled by a new RaftListener instead. I
> don't think we need the overhead of the MetadataImage + MetadataDelta for
> this feature since a RaftListener's `handleCommit()` and
> `handleLoadSnapshot()` contain `ClusterIdRecord`. However, this means
> needing a third listener besides the MetadataLoader and QuorumMetaListener,
> and therefore an additional call to log#read() when handling KRaft commits
> + snapshots. From my reading, it seems like the Kafka log layer does not
> attempt any caching, and instead we rely on the OS page cache. Because of
> this, I think we should be using MetadataPublisher, but let me know what
> you think.
>
> I am thinking of using an AtomicReference<String> to represent the
> clusterId in-memory. This RaftListener/MetadataPublisher will be the only
> writer to this value if it is not already defined by meta.properties, but
> there are many readers of this value. The initial value of this reference
> is null or the cluster.id from meta.properties. Upon reading
> `ClusterIdRecord`, the listener will throw an exception if it has a
> non-null clusterId and reads a ClusterIdRecord with a different ID. If it
> does not have cluster.id set and reads a ClusterIdRecord, it will update
> the AtomicReference and persist cluster.id to meta.properties. Let me know
> if this approach sounds reasonable to you.
>
> RE JS7: From what I understand about MetaPropertiesEnsemble and its
> verify() method, I think it is reasonable to say our
> RaftListener/MetadataPublisher will know how many (if any)
> `meta.properties` files it is responsible for persisting cluster.id to
> during the current process incarnation when it starts up. Currently we only
> validate the MetaPropertiesEnsemble in two places: during formatting, and
> during node startup. From what I understand, scenarios 1 and 2 should only
> occur alongside a restart of the kafka process (to generate a new
> directory-id and/or update log.dirs), but please correct me if this
> assumption is wrong. I'm not sure if scenario 3 is referring to a partial
> write of a given meta.properties (i.e. it does not contain cluster.id), or
> not writing the discovered cluster.id to all meta.properties files on the
> node before a crash. If a meta.properties does not exist in a log/metadata
> log directory during startup, we need to write a V2 one without a
> cluster.id,
> but we would be aware of this. If we succeed writing cluster.id to at
> least
> one meta.properties via the ClusterIdRecord, I believe it is safe to write
> that same value to the other meta.properties upon restart if they exist
> because cluster.id does not change.
>
> I may have previously removed this from the KIP, but given this discussion,
> I believe it is only safe to update the in-memory cluster.id only after
> writing this to all meta.properties on a node.
>
> RE JS8: Okay, maybe I will just rewrite the section. My point was to say
> something like: a node's discovery of the leader's committed cluster.id
> relies on the discovery of a HWM and our RaftListener/MetadataPublisher to
> be registered with the raft client, and that we need to wait for these
> things before the startup logic in Controller/BrokerServer executes.
> However, if our listener does not see the ClusterIdRecord in `handleCommit`
> or `handleLoadSnapshot`, it can't do anything meaningful, so it is more
> accurate to say we need to wait until ClusterIdRecord is committed.
>
> On Thu, Mar 19, 2026 at 12:57 AM José Armando García Sancio via dev <
> dev@kafka.apache.org> wrote:
>
> > Hi Kevin, Thanks for the KIP and excuse my delay response.
> >
> > JS1: Can you clarify that this KIP removes the need for all Kafka
> > nodes to be formatted pior to starting Kafka. However, this doesn't
> > prevent users from formatting their broker with a cluster ID if they
> > prefer. This is especially needed for Kafka nodes formatted for a
> > cluster using an MV that doesn't support this feature.
> >
> > JS2: How are you planning to implement "kafka-storage format
> > --clusterid YYY --standalone"? Is that going to behave like it does
> > today by writing the cluster id to the meta.properties files? Or are
> > you planning to write the cluster id using the ClusterIdRecord to the
> > bootstrap.checkpoint or 0-0.checkpoint (after KIP-1170)?
> >
> > JS3: In one of your replies you say "Discovering the cluster id value
> > for the first time would only require a single FetchSnapshot or a
> > Fetch of the bootstrap metadata records." This is not entirely
> > accurate. The best we can say is that brokers need to catch up to the
> > HWM before they can send a registration requests to the active
> > controller or it can start a few internal component. However, the
> > broker already had this requirement prior to this KIP, so it is not
> > new.
> >
> > JS4: In the KIP you mention "if meta.properties does not exist and the
> > node is a bootstrap controller, throw a runtime exception." Can you
> > explain how you plan to implement this? One important aspect to
> > consider is that in KRaft voters (controllers) are identified by the
> > node ID and directory ID. A node can recover from a disk failure by
> > coming back with the same node ID but a different directory ID. In
> > this case, the controller should auto-recover if the auto-join feature
> > is enabled.
> >
> > JS5: In the KIP you mention "One detail here is that observer
> > controllers with auto-join must wait until they have a cluster id
> > before trying to add or remove themselves." I understand the reason
> > for this requirement. If a node auto-joins the controller cluster, you
> > must guarantee that it knows the cluster id in case it becomes the
> > leader and needs to write the ClusterIDRecord. Can you elaborate on
> > your implementation plan?
> >
> > JS6: In the KIP you mention "This can be implemented as a
> > MetadataPublisher that registers to the raft client alongside the
> > MetadataLoader." Metadata publishers don't register with the KRaft
> > client. RaftClient.Listener register with the KRaft client. Metadata
> > publisher register with the metadata loader instead.
> >
> > JS7: One complexity is that there is a meta.properties per log
> > directory and metadata log directory. This means that in the stable
> > case the cluster ID exists in all the meta.properties files.
> > Unfortunately, this may not be the case for several reasons: 1) the
> > disk was replaced, 2) a new disk was added, or 3) the write operation
> > was only partially successful. How do you plan to handle this case?
> > Consider that the controller and the broker can run on the same JVM
> > and use a log directory different from the metadata log directory.
> > Controllers only read and write to the metadata log directory.
> >
> > JS8: In the KIP you mention "Learning of a HWM from the leader, which
> > the leader allows for because it will send valid fetch responses back
> > to nodes that do not have a cluster id." One implementation complexity
> > is that KRaft can discover the HWM and send a handleCommit event
> > without having fetched all data up to the HWM. What KRaft guarantees
> > is that the active leader will not receive a handleLeaderChange event
> > until it has caught up to the leader's epoch. How do you plan to
> > implement this?
> >
> > Thanks,
> > --
> > -José
> >
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi José,

Thanks for the replies and questions.

RE JS1: "Can you clarify that this KIP removes the need for all Kafka nodes
to be formatted prior to starting Kafka." Hmmm, I guess in the static
cluster case that skips formatting having a newer software version + older
MV is not a possible case, so I will remove that mention from the KIP. We
should default to the latest MV if we skip formatting, which will support
writing a ClusterIdRecord.

Right now, it is not completely clear to me how we can allow bootstrap
controllers (this applies mainly for kraft.version=0, since kraft.version=1
cannot elect a leader without proper formatting) to also skip formatting.
That is why I said in the proposed changes: "*Remove the requirement of
brokers and observer controllers to format before starting kafka"*. I agree
that KRaft can still elect a leader without clusterId in this case, but I'm
not completely sure how a QuorumController with an "empty" clusterId which
needs to be set later, should behave. My working idea is detailed in RE
JS6. This is required because the active controller needs to generate a
clusterId and write it back to KRaft upon activation in order for the
committed `ClusterIdRecord` to appear in records passed to
`RaftListener#handleCommit()`, so we cannot block its initialization.
Keeping the assumption that QuorumController.clusterId is final and
non-null would be nice, but that requires all KRaft voters to format with a
cluster.id. Let me know what you think about the best way to remove this
requirement.

RE JS2: My plan was to continue to write meta.properties V1 during
formatting with a `cluster.id` field like today, but also write a
`ClusterIdRecord` to the bootstrap snapshot for redundancy if the MV
supports it (I'm not sure if kafka is expected to handle only partial log
directory corruption/destruction). If the "bootstrap controller cluster id
check" from JS4 is correct, then the initial active controller is
guaranteed to have a non-null `cluster.id` in meta.properties. So long as
the MV supports it, the active controller would then write ClusterIdRecord
as part of the bootstrap records.

RE JS3: When I said this, I meant that the restriction of waiting for the
discovery of cluster.id to persist it to meta.properties during broker
startup is no more restrictive than what already currently exists, which is
being caught up to the HWM in order to register with the active controller.

RE JS 4: Yeah, I thought about this, specifically around the
kraft.version=1 case since it is less straightforward what a "bootstrap
controller" is. Under the current design, in kraft.version=0, any node who
is part of the `controller.quorum.voters` config must have
`meta.properties` with `cluster.id`. In kraft.version=1, any node who has a
`0-0.checkpoint` is considered a "bootstrap controller." This is a
heuristic, but I believe it is correct, since in order for the
0-0.checkpoint to not exist on a node which formatted with --standalone or
--initial-controllers, there must have either been another checkpoint with
committed records, which imply an elected initial leader, or a disk loss.
Whenever a voter with id X and initial directory-id A comes back as (X, B),
this process incarnation is an observer from the perspective of KRaft, and
I think we can assume it has neither `meta.properties` or `0-0.checkpoint`
if the operator did not format it (assumption from RE JS2 about the kinds
of storage failures we expect to handle are not partial directory
failures). In this case, the "bootstrap controller" check does not apply to
(X, B), and if auto-join is enabled, it will follow the steps detailed in
RE JS5 to recover and rejoin the voter set. If we remove the requirement on
all nodes to format, then we would not need to implement these checks.

RE JS5: An observer without clusterId who can auto-join will fetch until
its KafkaRaftClient updates the cluster id in-memory (basically, auto-join
is off until it discovers the leader's clusterId). If the observer has
clusterId, it needs to match the leader's to perform a successful fetch,
which is required for successfully adding a voter via auto-join.

RE JS6: Apologies, I meant to say a MetadataPublisher registered to the
MetadataLoader. Although, looking at this again, maybe this discovery +
persistence of clusterId can be handled by a new RaftListener instead. I
don't think we need the overhead of the MetadataImage + MetadataDelta for
this feature since a RaftListener's `handleCommit()` and
`handleLoadSnapshot()` contain `ClusterIdRecord`. However, this means
needing a third listener besides the MetadataLoader and QuorumMetaListener,
and therefore an additional call to log#read() when handling KRaft commits
+ snapshots. From my reading, it seems like the Kafka log layer does not
attempt any caching, and instead we rely on the OS page cache. Because of
this, I think we should be using MetadataPublisher, but let me know what
you think.

I am thinking of using an AtomicReference<String> to represent the
clusterId in-memory. This RaftListener/MetadataPublisher will be the only
writer to this value if it is not already defined by meta.properties, but
there are many readers of this value. The initial value of this reference
is null or the cluster.id from meta.properties. Upon reading
`ClusterIdRecord`, the listener will throw an exception if it has a
non-null clusterId and reads a ClusterIdRecord with a different ID. If it
does not have cluster.id set and reads a ClusterIdRecord, it will update
the AtomicReference and persist cluster.id to meta.properties. Let me know
if this approach sounds reasonable to you.

RE JS7: From what I understand about MetaPropertiesEnsemble and its
verify() method, I think it is reasonable to say our
RaftListener/MetadataPublisher will know how many (if any)
`meta.properties` files it is responsible for persisting cluster.id to
during the current process incarnation when it starts up. Currently we only
validate the MetaPropertiesEnsemble in two places: during formatting, and
during node startup. From what I understand, scenarios 1 and 2 should only
occur alongside a restart of the kafka process (to generate a new
directory-id and/or update log.dirs), but please correct me if this
assumption is wrong. I'm not sure if scenario 3 is referring to a partial
write of a given meta.properties (i.e. it does not contain cluster.id), or
not writing the discovered cluster.id to all meta.properties files on the
node before a crash. If a meta.properties does not exist in a log/metadata
log directory during startup, we need to write a V2 one without a cluster.id,
but we would be aware of this. If we succeed writing cluster.id to at least
one meta.properties via the ClusterIdRecord, I believe it is safe to write
that same value to the other meta.properties upon restart if they exist
because cluster.id does not change.

I may have previously removed this from the KIP, but given this discussion,
I believe it is only safe to update the in-memory cluster.id only after
writing this to all meta.properties on a node.

RE JS8: Okay, maybe I will just rewrite the section. My point was to say
something like: a node's discovery of the leader's committed cluster.id
relies on the discovery of a HWM and our RaftListener/MetadataPublisher to
be registered with the raft client, and that we need to wait for these
things before the startup logic in Controller/BrokerServer executes.
However, if our listener does not see the ClusterIdRecord in `handleCommit`
or `handleLoadSnapshot`, it can't do anything meaningful, so it is more
accurate to say we need to wait until ClusterIdRecord is committed.

On Thu, Mar 19, 2026 at 12:57 AM José Armando García Sancio via dev <
dev@kafka.apache.org> wrote:

> Hi Kevin, Thanks for the KIP and excuse my delay response.
>
> JS1: Can you clarify that this KIP removes the need for all Kafka
> nodes to be formatted pior to starting Kafka. However, this doesn't
> prevent users from formatting their broker with a cluster ID if they
> prefer. This is especially needed for Kafka nodes formatted for a
> cluster using an MV that doesn't support this feature.
>
> JS2: How are you planning to implement "kafka-storage format
> --clusterid YYY --standalone"? Is that going to behave like it does
> today by writing the cluster id to the meta.properties files? Or are
> you planning to write the cluster id using the ClusterIdRecord to the
> bootstrap.checkpoint or 0-0.checkpoint (after KIP-1170)?
>
> JS3: In one of your replies you say "Discovering the cluster id value
> for the first time would only require a single FetchSnapshot or a
> Fetch of the bootstrap metadata records." This is not entirely
> accurate. The best we can say is that brokers need to catch up to the
> HWM before they can send a registration requests to the active
> controller or it can start a few internal component. However, the
> broker already had this requirement prior to this KIP, so it is not
> new.
>
> JS4: In the KIP you mention "if meta.properties does not exist and the
> node is a bootstrap controller, throw a runtime exception." Can you
> explain how you plan to implement this? One important aspect to
> consider is that in KRaft voters (controllers) are identified by the
> node ID and directory ID. A node can recover from a disk failure by
> coming back with the same node ID but a different directory ID. In
> this case, the controller should auto-recover if the auto-join feature
> is enabled.
>
> JS5: In the KIP you mention "One detail here is that observer
> controllers with auto-join must wait until they have a cluster id
> before trying to add or remove themselves." I understand the reason
> for this requirement. If a node auto-joins the controller cluster, you
> must guarantee that it knows the cluster id in case it becomes the
> leader and needs to write the ClusterIDRecord. Can you elaborate on
> your implementation plan?
>
> JS6: In the KIP you mention "This can be implemented as a
> MetadataPublisher that registers to the raft client alongside the
> MetadataLoader." Metadata publishers don't register with the KRaft
> client. RaftClient.Listener register with the KRaft client. Metadata
> publisher register with the metadata loader instead.
>
> JS7: One complexity is that there is a meta.properties per log
> directory and metadata log directory. This means that in the stable
> case the cluster ID exists in all the meta.properties files.
> Unfortunately, this may not be the case for several reasons: 1) the
> disk was replaced, 2) a new disk was added, or 3) the write operation
> was only partially successful. How do you plan to handle this case?
> Consider that the controller and the broker can run on the same JVM
> and use a log directory different from the metadata log directory.
> Controllers only read and write to the metadata log directory.
>
> JS8: In the KIP you mention "Learning of a HWM from the leader, which
> the leader allows for because it will send valid fetch responses back
> to nodes that do not have a cluster id." One implementation complexity
> is that KRaft can discover the HWM and send a handleCommit event
> without having fetched all data up to the HWM. What KRaft guarantees
> is that the active leader will not receive a handleLeaderChange event
> until it has caught up to the leader's epoch. How do you plan to
> implement this?
>
> Thanks,
> --
> -José
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[José Armando García Sancio via dev <de...@kafka.apache.org>]

Hi Kevin, Thanks for the KIP and excuse my delay response.

JS1: Can you clarify that this KIP removes the need for all Kafka
nodes to be formatted pior to starting Kafka. However, this doesn't
prevent users from formatting their broker with a cluster ID if they
prefer. This is especially needed for Kafka nodes formatted for a
cluster using an MV that doesn't support this feature.

JS2: How are you planning to implement "kafka-storage format
--clusterid YYY --standalone"? Is that going to behave like it does
today by writing the cluster id to the meta.properties files? Or are
you planning to write the cluster id using the ClusterIdRecord to the
bootstrap.checkpoint or 0-0.checkpoint (after KIP-1170)?

JS3: In one of your replies you say "Discovering the cluster id value
for the first time would only require a single FetchSnapshot or a
Fetch of the bootstrap metadata records." This is not entirely
accurate. The best we can say is that brokers need to catch up to the
HWM before they can send a registration requests to the active
controller or it can start a few internal component. However, the
broker already had this requirement prior to this KIP, so it is not
new.

JS4: In the KIP you mention "if meta.properties does not exist and the
node is a bootstrap controller, throw a runtime exception." Can you
explain how you plan to implement this? One important aspect to
consider is that in KRaft voters (controllers) are identified by the
node ID and directory ID. A node can recover from a disk failure by
coming back with the same node ID but a different directory ID. In
this case, the controller should auto-recover if the auto-join feature
is enabled.

JS5: In the KIP you mention "One detail here is that observer
controllers with auto-join must wait until they have a cluster id
before trying to add or remove themselves." I understand the reason
for this requirement. If a node auto-joins the controller cluster, you
must guarantee that it knows the cluster id in case it becomes the
leader and needs to write the ClusterIDRecord. Can you elaborate on
your implementation plan?

JS6: In the KIP you mention "This can be implemented as a
MetadataPublisher that registers to the raft client alongside the
MetadataLoader." Metadata publishers don't register with the KRaft
client. RaftClient.Listener register with the KRaft client. Metadata
publisher register with the metadata loader instead.

JS7: One complexity is that there is a meta.properties per log
directory and metadata log directory. This means that in the stable
case the cluster ID exists in all the meta.properties files.
Unfortunately, this may not be the case for several reasons: 1) the
disk was replaced, 2) a new disk was added, or 3) the write operation
was only partially successful. How do you plan to handle this case?
Consider that the controller and the broker can run on the same JVM
and use a log directory different from the metadata log directory.
Controllers only read and write to the metadata log directory.

JS8: In the KIP you mention "Learning of a HWM from the leader, which
the leader allows for because it will send valid fetch responses back
to nodes that do not have a cluster id." One implementation complexity
is that KRaft can discover the HWM and send a handleCommit event
without having fetched all data up to the HWM. What KRaft guarantees
is that the active leader will not receive a handleLeaderChange event
until it has caught up to the leader's epoch. How do you plan to
implement this?

Thanks,
-- 
-José

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi José,

RE JS1: I have updated the KIP to discuss the formatting and KRaft
configuration requirements on a node level rather than a cluster level.

> Do you mean that to establish a KRaft leader, controller nodes must be
formatted with "kafka-storage format ... --standalone" or
"kafka-storage format ... --initial-controller" for a dynamic
controller cluster? And that the kraft controllers must specify the
controller.quorum.voters properties for a static controller quorum?

Yes, this is what I mean.

RE JS2: Sure, I think we can add this as part of this KIP, since it seems
straightforward. Upon discovery of the clusterId from the metadata, the
local node will upgrade any MPV0 or MPV1 to MPV2, and persist that to disk.

RE JS3: I have added this to the KIP.

Thanks for the discussions,
Kevin Wu

On Tue, Sep 22, 2026 at 12:22 PM José Armando García Sancio via dev <
dev@kafka.apache.org> wrote:

> Thanks for the responses.
>
> RE JS1:
>
> I am still having a hardtime understanding what we mean by cluster
> when we are discussing kafka-storage format. Kafka-storage format is a
> node operation not a cluster-wide operation.
>
> In your response, you have "In this case, `controller.quorum.voters`
> must be present to elect a KRaft leader. `controller.quorum.voters` is
> not required on nodes who skip formatting." Why not just say that?
> When explaining a distributed system I find it helpful to define the
> protocol in terms of individual nodes since the code implementing the
> protocol executes on individual nodes.
>
> To me the statement "Clusters who skip formatting will also have a
> static KRaft quorum, which is explained in the below section, so
> controller.quorum.voters must be defined. Operators can still end up
> with a dynamic quorum without formatting because static to dynamic
> quorum upgrades are supported." is not really implementable.
>
> Do you mean that to establish a KRaft leader, controller nodes must be
> formatted with "kafka-storage format ... --standalone" or
> "kafka-storage format ... --initial-controller" for a dynamic
> controller cluster? And that the kraft controllers must specify the
> controller.quorum.voters properties for a static controller quorum?
>
>
> RE JS2:
> "For clusters with MV >= 150, it is fine for
> a given node to have MPV1 because it was previously formatted (this
> includes all existing clusters), but if a given node was not formatted, it
> will have MPV2."
>
> It is fine for the node to be in meta.properties version 1 but the
> question is whether we should upgrade the meta.properties version to 2
> if the MV is upgraded to 150 in your example. It is good practice to
> limit the use of older versions so that eventually we can deprecate
> them.
>
> RE JS3:
>
> Sounds good.
>
> On Mon, Sep 21, 2026 at 3:00 PM Kevin Wu <ke...@gmail.com> wrote:
> >
> > Hi José,
> >
> > Thanks for the questions.
> >
> > RE JS1:
> > I think my language may have been a bit confusing. What I mean by
> "clusters
> > who skip formatting" is a cluster in which no node (controller or broker)
> > invokes `kafka-storage format` before starting kafka. In this case,
> > `controller.quorum.voters` must be present to elect a KRaft leader.
> > `controller.quorum.voters` is not required on nodes who skip formatting.
> >
> > For example, to startup a new KIP-853 cluster with auto-formatting, the
> > operator could format one node with `--standalone`, and then skip
> > formatting on all other nodes. If the operator wants to add a new
> observer
> > controller to this existing KIP-853 cluster, the operator can also skip
> > formatting that node, and that node does not need
> > `controller.quorum.voters` so long as
> `controller.quorum.bootstrap.servers`
> > is defined to discover the quorum. I explained some of these things in
> this
> > section section:
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406618510/KIP-1262*Enable*auto-formatting*directories*KIP1262*3AEnableautoformattingdirectories-Whenisformattingstillrequiredbeforestartingkafka__;KysrIyU!!Ayb5sqE7!rIR_OnksedyKJKLdEHtQGHmBmhnGtMq3ZIkdfj1x5p9a-6QAv2r6PvvULfHrixiKqADTcReYdd1r9QkACZKHDG7G$
> .
> > I can add some of the discussion here to that section of the KIP. Does
> this
> > answer your questions?
> >
> > For now, skipping formatting entirely for a new KIP-853 cluster is out of
> > scope. See RE JR12 for details as to why.
> >
> > RE JS2:
> > I am not sure if I agree that a cluster's MV determines which version of
> > meta.properties to write. I believe only one thing should determine the
> > meta.properties version (MPV), which is whether the operator formatted
> the
> > local node. Assume the MV this feature introduces is MV 150. The
> > auto-formatting feature works with both MPV1 and MPV2. The MPV1 case is
> > less interesting because it means the clusterId is already known at
> format
> > time.
> >
> > When the operator invokes `kafka-storage format`, MPV 1 is
> > written, regardless of the MV.
> >
> > On the other hand, if the operator skips formatting on any node, the only
> > valid cluster MVs are necessarily the ones that support
> `ClusterIdRecord`,
> > i.e. MV >= 150. For clusters with MV < 150, brokers and controllers that
> > skip formatting will fail to complete startup, since there is no way for
> > them to discover the clusterId, and ideally, we should report this
> > misconfiguration in the error. For clusters with MV >= 150, it is fine
> for
> > a given node to have MPV1 because it was previously formatted (this
> > includes all existing clusters), but if a given node was not formatted,
> it
> > will have MPV2.
> >
> > Because this feature is gated behind a finalized MV 150, we do not have
> to
> > worry about MPV backwards compatibility, because it is not a valid
> > operation to downgrade the software version of a cluster with a finalized
> > MV 150 to a software version that does not support MV 150.
> >
> > RE JS3:
> > The high level idea was to block broker startup on discovering the
> > clusterId, since broker registration, which also blocks startup and
> > requires the broker to catch up with the metadata, relies on the
> clusterId.
> > On the broker, this means Kafka will wait for `
> server.max.startup.time.ms`
> > to discover the clusterId. I can include more details about this in this
> > section:
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406618510/KIP-1262*Enable*auto-formatting*directories*KIP1262*3AEnableautoformattingdirectories-CompletionofBrokerServerandControllerServerstartupmustblockonthediscoveryandpersistenceofthecluster.idtometa.properties__;KysrIyU!!Ayb5sqE7!rIR_OnksedyKJKLdEHtQGHmBmhnGtMq3ZIkdfj1x5p9a-6QAv2r6PvvULfHrixiKqADTcReYdd1r9QkACbJaNL6S$
> >
> > On the controller, the controller also waits for `
> server.max.startup.time.ms`
> > to finish startup. I believe it is fine to wait for the clusterId to be
> > committed after the existing blocking for authorizer futures to complete.
> >
> > Best,
> > Kevin Wu
> >
> > On Mon, Sep 21, 2026 at 12:48 PM José Armando García Sancio via dev <
> > dev@kafka.apache.org> wrote:
> >
> > > Hi Kevin,
> > >
> > > I was able to take a look at the KIP today.
> > >
> > > JS1:
> > >
> > > In the
> > > > Clusters who skip formatting will also have a static KRaft quorum,
> which
> > > is explained in the below section, so controller.quorum.voters must be
> > > defined. Operators can still end up with a dynamic quorum without
> > > formatting because static to dynamic quorum upgrades are supported.
> > >
> > > What do you mean by "Clusters who skip formatting"? Clusters are not
> > > formatted. Users format nodes instead. Why is controller.quorum.voters
> > > required if a node skips formatting? If this is a requirement, how
> > > would the user provision a new controller (observer) node to join an
> > > existing cluster through KIP-853? The controller.quorum.voters
> > > properties should not be required in a node that will join a dynamic
> > > controller cluster.
> > >
> > > JS2:
> > >
> > > When to write version 2 of the meta.properties file? I am trying to
> > > understand when Kafka will write version 2 of the meta.properties
> > > file. I would assume that this is based on the MV but I don't see this
> > > stated explicitly in the KIP. When the MV changes to a version that
> > > supports meta.properties version 2 will Kafka automatically upgrade
> > > the version even if there are no changes to the state stored in
> > > meta.properties?
> > >
> > > JS3:
> > >
> > > How long will Kafka wait for the cluster id to be discovered? What
> > > happens if the cluster id is not in the cluster metadata partition and
> > > all of the meta.properties files?
> > >
> > > Thanks,
> > > --
> > > -José
> > >
>
>
>
> --
> -José
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[José Armando García Sancio via dev <de...@kafka.apache.org>]

Thanks for the responses.

RE JS1:

I am still having a hardtime understanding what we mean by cluster
when we are discussing kafka-storage format. Kafka-storage format is a
node operation not a cluster-wide operation.

In your response, you have "In this case, `controller.quorum.voters`
must be present to elect a KRaft leader. `controller.quorum.voters` is
not required on nodes who skip formatting." Why not just say that?
When explaining a distributed system I find it helpful to define the
protocol in terms of individual nodes since the code implementing the
protocol executes on individual nodes.

To me the statement "Clusters who skip formatting will also have a
static KRaft quorum, which is explained in the below section, so
controller.quorum.voters must be defined. Operators can still end up
with a dynamic quorum without formatting because static to dynamic
quorum upgrades are supported." is not really implementable.

Do you mean that to establish a KRaft leader, controller nodes must be
formatted with "kafka-storage format ... --standalone" or
"kafka-storage format ... --initial-controller" for a dynamic
controller cluster? And that the kraft controllers must specify the
controller.quorum.voters properties for a static controller quorum?


RE JS2:
"For clusters with MV >= 150, it is fine for
a given node to have MPV1 because it was previously formatted (this
includes all existing clusters), but if a given node was not formatted, it
will have MPV2."

It is fine for the node to be in meta.properties version 1 but the
question is whether we should upgrade the meta.properties version to 2
if the MV is upgraded to 150 in your example. It is good practice to
limit the use of older versions so that eventually we can deprecate
them.

RE JS3:

Sounds good.

On Mon, Sep 21, 2026 at 3:00 PM Kevin Wu <ke...@gmail.com> wrote:
>
> Hi José,
>
> Thanks for the questions.
>
> RE JS1:
> I think my language may have been a bit confusing. What I mean by "clusters
> who skip formatting" is a cluster in which no node (controller or broker)
> invokes `kafka-storage format` before starting kafka. In this case,
> `controller.quorum.voters` must be present to elect a KRaft leader.
> `controller.quorum.voters` is not required on nodes who skip formatting.
>
> For example, to startup a new KIP-853 cluster with auto-formatting, the
> operator could format one node with `--standalone`, and then skip
> formatting on all other nodes. If the operator wants to add a new observer
> controller to this existing KIP-853 cluster, the operator can also skip
> formatting that node, and that node does not need
> `controller.quorum.voters` so long as `controller.quorum.bootstrap.servers`
> is defined to discover the quorum. I explained some of these things in this
> section section:
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406618510/KIP-1262*Enable*auto-formatting*directories*KIP1262*3AEnableautoformattingdirectories-Whenisformattingstillrequiredbeforestartingkafka__;KysrIyU!!Ayb5sqE7!rIR_OnksedyKJKLdEHtQGHmBmhnGtMq3ZIkdfj1x5p9a-6QAv2r6PvvULfHrixiKqADTcReYdd1r9QkACZKHDG7G$ .
> I can add some of the discussion here to that section of the KIP. Does this
> answer your questions?
>
> For now, skipping formatting entirely for a new KIP-853 cluster is out of
> scope. See RE JR12 for details as to why.
>
> RE JS2:
> I am not sure if I agree that a cluster's MV determines which version of
> meta.properties to write. I believe only one thing should determine the
> meta.properties version (MPV), which is whether the operator formatted the
> local node. Assume the MV this feature introduces is MV 150. The
> auto-formatting feature works with both MPV1 and MPV2. The MPV1 case is
> less interesting because it means the clusterId is already known at format
> time.
>
> When the operator invokes `kafka-storage format`, MPV 1 is
> written, regardless of the MV.
>
> On the other hand, if the operator skips formatting on any node, the only
> valid cluster MVs are necessarily the ones that support `ClusterIdRecord`,
> i.e. MV >= 150. For clusters with MV < 150, brokers and controllers that
> skip formatting will fail to complete startup, since there is no way for
> them to discover the clusterId, and ideally, we should report this
> misconfiguration in the error. For clusters with MV >= 150, it is fine for
> a given node to have MPV1 because it was previously formatted (this
> includes all existing clusters), but if a given node was not formatted, it
> will have MPV2.
>
> Because this feature is gated behind a finalized MV 150, we do not have to
> worry about MPV backwards compatibility, because it is not a valid
> operation to downgrade the software version of a cluster with a finalized
> MV 150 to a software version that does not support MV 150.
>
> RE JS3:
> The high level idea was to block broker startup on discovering the
> clusterId, since broker registration, which also blocks startup and
> requires the broker to catch up with the metadata, relies on the clusterId.
> On the broker, this means Kafka will wait for `server.max.startup.time.ms`
> to discover the clusterId. I can include more details about this in this
> section:
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406618510/KIP-1262*Enable*auto-formatting*directories*KIP1262*3AEnableautoformattingdirectories-CompletionofBrokerServerandControllerServerstartupmustblockonthediscoveryandpersistenceofthecluster.idtometa.properties__;KysrIyU!!Ayb5sqE7!rIR_OnksedyKJKLdEHtQGHmBmhnGtMq3ZIkdfj1x5p9a-6QAv2r6PvvULfHrixiKqADTcReYdd1r9QkACbJaNL6S$
>
> On the controller, the controller also waits for `server.max.startup.time.ms`
> to finish startup. I believe it is fine to wait for the clusterId to be
> committed after the existing blocking for authorizer futures to complete.
>
> Best,
> Kevin Wu
>
> On Mon, Sep 21, 2026 at 12:48 PM José Armando García Sancio via dev <
> dev@kafka.apache.org> wrote:
>
> > Hi Kevin,
> >
> > I was able to take a look at the KIP today.
> >
> > JS1:
> >
> > In the
> > > Clusters who skip formatting will also have a static KRaft quorum, which
> > is explained in the below section, so controller.quorum.voters must be
> > defined. Operators can still end up with a dynamic quorum without
> > formatting because static to dynamic quorum upgrades are supported.
> >
> > What do you mean by "Clusters who skip formatting"? Clusters are not
> > formatted. Users format nodes instead. Why is controller.quorum.voters
> > required if a node skips formatting? If this is a requirement, how
> > would the user provision a new controller (observer) node to join an
> > existing cluster through KIP-853? The controller.quorum.voters
> > properties should not be required in a node that will join a dynamic
> > controller cluster.
> >
> > JS2:
> >
> > When to write version 2 of the meta.properties file? I am trying to
> > understand when Kafka will write version 2 of the meta.properties
> > file. I would assume that this is based on the MV but I don't see this
> > stated explicitly in the KIP. When the MV changes to a version that
> > supports meta.properties version 2 will Kafka automatically upgrade
> > the version even if there are no changes to the state stored in
> > meta.properties?
> >
> > JS3:
> >
> > How long will Kafka wait for the cluster id to be discovered? What
> > happens if the cluster id is not in the cluster metadata partition and
> > all of the meta.properties files?
> >
> > Thanks,
> > --
> > -José
> >



-- 
-José

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[Kevin Wu <ke...@gmail.com>]

Hi José,

Thanks for the questions.

RE JS1:
I think my language may have been a bit confusing. What I mean by "clusters
who skip formatting" is a cluster in which no node (controller or broker)
invokes `kafka-storage format` before starting kafka. In this case,
`controller.quorum.voters` must be present to elect a KRaft leader.
`controller.quorum.voters` is not required on nodes who skip formatting.

For example, to startup a new KIP-853 cluster with auto-formatting, the
operator could format one node with `--standalone`, and then skip
formatting on all other nodes. If the operator wants to add a new observer
controller to this existing KIP-853 cluster, the operator can also skip
formatting that node, and that node does not need
`controller.quorum.voters` so long as `controller.quorum.bootstrap.servers`
is defined to discover the quorum. I explained some of these things in this
section section:
https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406618510/KIP-1262+Enable+auto-formatting+directories#KIP1262%3AEnableautoformattingdirectories-Whenisformattingstillrequiredbeforestartingkafka.
I can add some of the discussion here to that section of the KIP. Does this
answer your questions?

For now, skipping formatting entirely for a new KIP-853 cluster is out of
scope. See RE JR12 for details as to why.

RE JS2:
I am not sure if I agree that a cluster's MV determines which version of
meta.properties to write. I believe only one thing should determine the
meta.properties version (MPV), which is whether the operator formatted the
local node. Assume the MV this feature introduces is MV 150. The
auto-formatting feature works with both MPV1 and MPV2. The MPV1 case is
less interesting because it means the clusterId is already known at format
time.

When the operator invokes `kafka-storage format`, MPV 1 is
written, regardless of the MV.

On the other hand, if the operator skips formatting on any node, the only
valid cluster MVs are necessarily the ones that support `ClusterIdRecord`,
i.e. MV >= 150. For clusters with MV < 150, brokers and controllers that
skip formatting will fail to complete startup, since there is no way for
them to discover the clusterId, and ideally, we should report this
misconfiguration in the error. For clusters with MV >= 150, it is fine for
a given node to have MPV1 because it was previously formatted (this
includes all existing clusters), but if a given node was not formatted, it
will have MPV2.

Because this feature is gated behind a finalized MV 150, we do not have to
worry about MPV backwards compatibility, because it is not a valid
operation to downgrade the software version of a cluster with a finalized
MV 150 to a software version that does not support MV 150.

RE JS3:
The high level idea was to block broker startup on discovering the
clusterId, since broker registration, which also blocks startup and
requires the broker to catch up with the metadata, relies on the clusterId.
On the broker, this means Kafka will wait for `server.max.startup.time.ms`
to discover the clusterId. I can include more details about this in this
section:
https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406618510/KIP-1262+Enable+auto-formatting+directories#KIP1262%3AEnableautoformattingdirectories-CompletionofBrokerServerandControllerServerstartupmustblockonthediscoveryandpersistenceofthecluster.idtometa.properties

On the controller, the controller also waits for `server.max.startup.time.ms`
to finish startup. I believe it is fine to wait for the clusterId to be
committed after the existing blocking for authorizer futures to complete.

Best,
Kevin Wu

On Mon, Sep 21, 2026 at 12:48 PM José Armando García Sancio via dev <
dev@kafka.apache.org> wrote:

> Hi Kevin,
>
> I was able to take a look at the KIP today.
>
> JS1:
>
> In the
> > Clusters who skip formatting will also have a static KRaft quorum, which
> is explained in the below section, so controller.quorum.voters must be
> defined. Operators can still end up with a dynamic quorum without
> formatting because static to dynamic quorum upgrades are supported.
>
> What do you mean by "Clusters who skip formatting"? Clusters are not
> formatted. Users format nodes instead. Why is controller.quorum.voters
> required if a node skips formatting? If this is a requirement, how
> would the user provision a new controller (observer) node to join an
> existing cluster through KIP-853? The controller.quorum.voters
> properties should not be required in a node that will join a dynamic
> controller cluster.
>
> JS2:
>
> When to write version 2 of the meta.properties file? I am trying to
> understand when Kafka will write version 2 of the meta.properties
> file. I would assume that this is based on the MV but I don't see this
> stated explicitly in the KIP. When the MV changes to a version that
> supports meta.properties version 2 will Kafka automatically upgrade
> the version even if there are no changes to the state stored in
> meta.properties?
>
> JS3:
>
> How long will Kafka wait for the cluster id to be discovered? What
> happens if the cluster id is not in the cluster metadata partition and
> all of the meta.properties files?
>
> Thanks,
> --
> -José
>

Re: [DISCUSS] KIP-1262: Enable auto-formatting directories

[José Armando García Sancio via dev <de...@kafka.apache.org>]

Hi Kevin,

I was able to take a look at the KIP today.

JS1:

In the
> Clusters who skip formatting will also have a static KRaft quorum, which is explained in the below section, so controller.quorum.voters must be defined. Operators can still end up with a dynamic quorum without formatting because static to dynamic quorum upgrades are supported.

What do you mean by "Clusters who skip formatting"? Clusters are not
formatted. Users format nodes instead. Why is controller.quorum.voters
required if a node skips formatting? If this is a requirement, how
would the user provision a new controller (observer) node to join an
existing cluster through KIP-853? The controller.quorum.voters
properties should not be required in a node that will join a dynamic
controller cluster.

JS2:

When to write version 2 of the meta.properties file? I am trying to
understand when Kafka will write version 2 of the meta.properties
file. I would assume that this is based on the MV but I don't see this
stated explicitly in the KIP. When the MV changes to a version that
supports meta.properties version 2 will Kafka automatically upgrade
the version even if there are no changes to the state stored in
meta.properties?

JS3:

How long will Kafka wait for the cluster id to be discovered? What
happens if the cluster id is not in the cluster metadata partition and
all of the meta.properties files?

Thanks,
-- 
-José